Move most of AddBuilds to hydra-eval-jobset

[?]

Feb 21, 2017, 3:17 PM

VU2OLHD246DFWL2WPFSKWPLXTN3WRL25TSRU4PJMJBNWDF35OFRAC

Dependencies

[2] MSDSOL5L Fix same system inputs
[3] ZTQ5WWHD Pass project and jobset to fetchInput
[4] E3IEH57U Revert "Make evaluation fail with proper error when a input of type build is not available."
[5] GREPCKGI Die tabs die
[6] 5KWIPCUK UTF-8 fix
[7] O2BEFBIO buildInputToString: Use inputType attribute instead of type attribute
[8] GXKMNJ26 Pass a build's drv path as a store path
[9] BGDZAEU7 Remove unneeded camelcase
[10] KLFSX2R6 hydra-evaluator: Fix input change check
[11] 5TZGO7RJ Store unset descriptions etc. as nulls
[12] 53IMJNBB Add isChannel column and meta attribute.
[13] I6QMKSIS Move getBaseUrl
[14] DKIA7GAD Better fix for dots in jobset names
[15] NY77R3JJ Bump evaluation timeout to 6 hours
[16] MPFSVI5X Pedantry: CLOCK_REALTIME -> CLOCK_MONOTONIC
[17] 4FWDVNWA Pass additional attributes for Git inputs
[18] LBNVQXUB * Build the /build stuff in a separate controller.
[19] PXUCXYZI * Pass `-j 1' to hydra_eval_jobs to ensure that it can make progress
[20] ARD6Z67T Do incremental SVN checkouts
[21] JAH3UPWA Support revision control systems via plugins
[22] TPNHTE5V Remove obsolete Builds columns and provide accurate "Running builds"
[23] 7DWCXNC7 Use the new Nix Perl bindings
[24] IMQRX4MP hydra-eval-jobs: Use JSON instead of XML
[25] BKOIYITR added some json responses
[26] H7CNGK4O * Log evaluation errors etc. in the DB.
[27] P5XCKTFD Fix sysbuild input type handling
[28] FM4O2L4M hydra: if evaluator sees cached build, also add the buildproducts
[29] OOQ2D3KC * Refactoring: move fetchInput out of hydra_scheduler into a separate
[30] PGSSRA7C Add an input type "nix" for passing arbitrary Nix expressions
[31] HPEG2RHV Merge the BuildResultInfo table into the Builds table
[32] Y6FFX4YB Make evaluation fail with proper error when a input of type build is not available.
[33] OSVLMLCQ hydra: factored out build restart and
[34] E7M2WP7A Remove unused Jobs columns
[35] Z52T2BC4 Support passing a jobset evaluation as an input
[36] 6GJSWZ4F Copy-paste error
[37] JYZEGF56 Create Builds with iscurrent set
[38] RWIBJ5L4 * Autoflush stdout.
[39] ZWCTAZGL added newsitems, added some admin options to clear various caches.
[40] X27GNHDV * Basic job info in the database.
[41] LZVO64YG Merge in the first bits of the API work
[42] SM5M2J3A Pass inputs to release expressions using -I
[43] JOVVHIJX Remove all entry points to modify machines
[44] FAIJDQKZ
[45] KHYZVPBR Propagate checkresponsible from JobsetInput to BuildInput
[46] 5NO7NCKT * Refactoring.
[47] RFE6T5LG * Store jobset evaluations in the database explicitly. This includes
[48] DDGBLKEN Update isCurrent properly
[49] NFVN7JRB Handle missing "build" inputs
[50] IEXUBVNB allow dots (.) in job names when used as build input
[51] A43SLRSH Fix handling of IPC::Run::run exit status
[52] 6WRGCITD Enable declarative projects.
[53] WWUOQ7V4 * hydra: indentation and fixed duplicate key in cachedgitinputs bug
[54] R4MHON3O pass svn/bzr revisions as integers
[55] JOYONH2K Prevent multiple builds with the same (job, outPath) tuple from being added
[56] DVLDHOUZ hydra-evaluator: Reduce verbosity
[57] QMW24O5S Add support for Guile & Guix.
[58] RXVJFQ5A Evaluator cleanups
[59] SHYRGAWZ hydra: when no external url is given for diff's of git/hg input, revert to a diff done on a local available clone
[60] SS4TZXNU Distinguish between permanent evaluation errors and transient input errors
[61] 4OXJRURP buildInputToString: Pass along the input's type and urr
[62] JZE7DC2F Whitespace
[63] FXW2UR7F initial bzr support
[64] WVD3YYON hydra-evaluator: Add some debug code
[65] HE3GX5IP Optimize fetch-git.
[66] Q24QXGSM * Don't do pretty printing for large logs, because the XSLT processing
[67] 5SHCWE7X * Prevent repeated evaluation of a jobset with the same inputs. This
[68] PIMGMGAF Rename hydra_eval_jobs to hydra-eval-jobs
[69] MRJCQ4EO Don't use given/when
[70] MOX7XJ2E Merge the BuildSchedulingInfo table into the Builds table
[71] H3S3H752 Pass along drvPath and outputName for inputs that are previous builds.
[72] INNOEHO6 * Fix getBuildLog for bzip2'd files.
[73] SMCOU72F hydra: add some admin for adding/enabling/etc build machines
[74] ZDENAYQI * email notification of evaluation errors to project owner (if desired)
[75] QT4FO2HP refactored admin controller, using chains, to avoid using requireadmin on each endpoint
[76] CHQEG6WY Hydra/29: Added timeout to svn/git checkouts, job evaluation
[77] CPMIKBDT Allow dots in job specifier of input type 'Previous build'
[78] 5GPK54IV Log segfaults from the evaluator
[79] DDMYFZ5X Fix the jobset unchanged check
[80] XDDCO6CH * hydra: add dependency list with links to logfiles
[81] NP4OLR5T Don't pass an undefined input
[82] QW7GTTXI Don't maintain BuildInputs anymore
[83] AFTXA575 * $HYDRA_DATA environment variable.
[84] 3XTHEUMP * Implemented the clone feature.
[85] HX4QYOYA add first evaluations tests
[86] POPU2ATH * hydra_scheduler: use eval-jobs.
[87] AKRVETP5 Handle UTF-8 characters in eval error messages
[88] A22P7HCO hydra: at evaluation, check if path is already built, and mark as built in stead of adding to the queue.
[89] RX5IIZMT Use Email::MIME instead of Email::Simple
[90] 5SMQ2PLK Fix tests
[91] JK2QWPH6
[92] QS4OX6Z7 Huuuge speedup in the Hydra evaluator
[93] JCJJKRWQ Handle job aliases in AggregateConstituents
[94] FYWE74AA
[95] X5EOJLNA Allow passing a specific build as an input
[96] NS7SND6R hydra-evaluator: Send statistics to statsd
[97] N22GPKYT * Put info about logs / build products in the DB.
[98] 2WRTOU2Z Cleanup
[99] BMSQD2ZH Indentation
[100] PMNWRTGJ Add multiple output support
[101] K3HODXGH Check all inputs for blame but only email selected inputs
[102] YFPZ46YK * hydra: added variant of build input type, 'build output (same system)' to allow better continous integration in one jobset for multiple system. it makes sure that the system of the build that is passed as input for a job has the same system as the job.
[103] VHV6GI4L Add a jobset eval action to restart all aborted/cancelled builds
[104] D3YETHSB Only pass the drv path if it is still valid
[105] SKQXOQ7T hydra: fixed missing argument to restartbuild function
[106] PHNLYPKB Call buildFinished when a cached build is added
[107] TJK27WSB Open the DB using Hydra::Model::DB->new
[108] EDRUQ4UK Die TABs die
[109] FTPCV25M Store aggregate members in the database
[110] KQS7DSKJ * Clean up indentation.
[111] JTRG7RDQ add support for git as jobinput
[112] ZTQEU5QS Hydra: Add support for maxSilent meta attribute (also already added timeout, but not implemented the actual timeout for the build yet)
[113] 24BMQDZA Start of single-process hydra-queue-runner
[114] SGNXIOI4 Hydra/32: Add option to force evaluation of a certain jobset via web interface (for admins only)
[115] CQTN62OH Die tabs die
[*] 2GK5DOU7 * Downloading closures.

Change contents

edit in src/lib/Hydra/Controller/API.pm at line 8

[17.141]→[17.0:30](∅→∅)

use Hydra::Helper::AddBuilds;
edit in src/lib/Hydra/Controller/Admin.pm at line 8

[17.154]→[17.0:30](∅→∅)

use Hydra::Helper::AddBuilds;
edit in src/lib/Hydra/Controller/Build.pm at line 9

[17.156]→[17.0:30](∅→∅)

use Hydra::Helper::AddBuilds;
edit in src/lib/Hydra/Helper/AddBuilds.pm at line 7

[17.1945]→[17.1290:1304](∅→∅),[17.2735]→[17.1290:1304](∅→∅)

use IPC::Run;
replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 22

[17.18]→[17.576:624](∅→∅),[17.624]→[17.710:769](∅→∅)

fetchInput evalJobs checkBuild inputsToArgs
restartBuild getPrevJobsetEval updateDeclarativeJobset

[17.18]

[17.769]

updateDeclarativeJobset
edit in src/lib/Hydra/Helper/AddBuilds.pm at line 25

[17.3175]→[17.0:1](∅→∅),[17.1]→[17.3175:3176](∅→∅),[17.3175]→[17.3175:3176](∅→∅),[17.3]→[17.3177:3502](∅→∅),[17.3177]→[17.3177:3502](∅→∅),[17.3502]→[14.0:82](∅→∅),[17.68]→[17.3568:4413](∅→∅),[17.70]→[17.3568:4413](∅→∅),[14.82]→[17.3568:4413](∅→∅),[17.3568]→[17.3568:4413](∅→∅),[17.3727]→[17.262:265](∅→∅),[17.8701]→[17.262:265](∅→∅),[17.265]→[17.3728:3729](∅→∅),[17.3729]→[17.265:287](∅→∅),[17.265]→[17.265:287](∅→∅),[17.287]→[17.165:218](∅→∅),[17.218]→[17.8707:8708](∅→∅),[17.347]→[17.8707:8708](∅→∅),[17.8707]→[17.8707:8708](∅→∅),[17.8708]→[17.110:129](∅→∅),[17.129]→[17.8900:8901](∅→∅),[17.3879]→[17.8900:8901](∅→∅),[17.8900]→[17.8900:8901](∅→∅),[17.8901]→[17.130:783](∅→∅),[17.783]→[17.9287:9288](∅→∅),[17.4242]→[17.9287:9288](∅→∅),[17.9287]→[17.9287:9288](∅→∅),[17.9288]→[4.0:87](∅→∅),[4.87]→[17.9475:9476](∅→∅),[17.87]→[17.9475:9476](∅→∅),[17.126]→[17.9475:9476](∅→∅),[17.4414]→[17.9475:9476](∅→∅),[17.9475]→[17.9475:9476](∅→∅),[17.9476]→[17.4415:4492](∅→∅),[17.4492]→[17.9557:9558](∅→∅),[17.9557]→[17.9557:9558](∅→∅),[17.9558]→[17.4493:4593](∅→∅),[17.4593]→[17.9666:9667](∅→∅),[17.9666]→[17.9666:9667](∅→∅),[17.9667]→[17.4477:4545](∅→∅),[17.4545]→[17.4674:4741](∅→∅),[17.4674]→[17.4674:4741](∅→∅),[17.4741]→[17.0:49](∅→∅),[17.49]→[17.222:223](∅→∅),[17.4741]→[17.222:223](∅→∅),[17.223]→[17.0:17](∅→∅),[17.17]→[17.50:91](∅→∅),[17.234]→[17.50:91](∅→∅),[17.91]→[17.4823:4884](∅→∅),[17.262]→[17.4823:4884](∅→∅),[17.4807]→[17.4823:4884](∅→∅),[17.4823]→[17.4823:4884](∅→∅),[17.4884]→[17.92:134](∅→∅),[17.175]→[17.4884:4895](∅→∅),[17.4884]→[17.4884:4895](∅→∅),[17.4895]→[9.0:94](∅→∅),[17.50]→[17.111:138](∅→∅),[9.94]→[17.111:138](∅→∅),[17.111]→[17.111:138](∅→∅),[17.138]→[17.134:136](∅→∅),[17.4895]→[17.134:136](∅→∅),[17.10008]→[17.134:136](∅→∅),[17.136]→[17.4896:4897](∅→∅),[17.4897]→[17.136:165](∅→∅),[17.136]→[17.136:165](∅→∅),[17.165]→[17.263:316](∅→∅),[17.316]→[17.225:226](∅→∅),[17.225]→[17.225:226](∅→∅),[17.226]→[17.4898:5047](∅→∅),[17.5047]→[17.387:388](∅→∅),[17.387]→[17.387:388](∅→∅),[17.388]→[17.5048:5182](∅→∅),[17.5182]→[17.528:529](∅→∅),[17.528]→[17.528:529](∅→∅),[17.529]→[17.5183:5249](∅→∅),[17.5249]→[2.0:80](∅→∅),[2.80]→[17.5317:5323](∅→∅),[17.4889]→[17.5317:5323](∅→∅),[17.5317]→[17.5317:5323](∅→∅)

sub parseJobName {
# Parse a job specification of the form `<project>:<jobset>:<job>
# [attrs]'. The project, jobset and attrs may be omitted. The
# attrs have the form `name = "value"'.
my ($s) = @_;
our $key;
our %attrs = ();
# hm, maybe I should stop programming Perl before it's too late...
$s =~ / ^ (?: (?: ($projectNameRE) : )? ($jobsetNameRE) : )? ($jobNameRE) \s*
(\[ \s* (
([\w]+) (?{ $key = $^N; }) \s* = \s* \"
([\w\-]+) (?{ $attrs{$key} = $^N; }) \"
\s* )* \])? $
/x
or die "invalid job specifier `$s'";
return ($1, $2, $3, \%attrs);
}

sub attrsToSQL {
my ($attrs, $id) = @_;

my $query = "1 = 1";

foreach my $name (keys %{$attrs}) {
my $value = $attrs->{$name};
$name =~ /^[\w\-]+$/ or die;
$value =~ /^[\w\-]+$/ or die;
# !!! Yes, this is horribly injection-prone... (though
# name/value are filtered above). Should use SQL::Abstract,
# but it can't deal with subqueries. At least we should use
# placeholders.
$query .= " and exists (select 1 from buildinputs where build = $id and name = '$name' and value = '$value')";
}

return $query;
}

sub fetchInputBuild {
my ($db, $project, $jobset, $name, $value) = @_;

my $prevBuild;

if ($value =~ /^\d+$/) {
$prevBuild = $db->resultset('Builds')->find({ id => int($value) });
} else {
my ($projectName, $jobsetName, $jobName, $attrs) = parseJobName($value);
$projectName ||= $project->name;
$jobsetName ||= $jobset->name;

# Pick the most recent successful build of the specified job.
$prevBuild = $db->resultset('Builds')->search(
{ finished => 1, project => $projectName, jobset => $jobsetName
, job => $jobName, buildStatus => 0 },
{ order_by => "me.id DESC", rows => 1
, where => \ attrsToSQL($attrs, "me.id") })->single;
}

return () if !defined $prevBuild || !isValidPath(getMainOutput($prevBuild)->path);

#print STDERR "input `", $name, "': using build ", $prevBuild->id, "\n";

my $pkgNameRE = "(?:(?:[A-Za-z0-9]|(?:-[^0-9]))+)";
my $versionRE = "(?:[A-Za-z0-9\.\-]+)";

my $relName = ($prevBuild->releasename or $prevBuild->nixname);
my $version = $2 if $relName =~ /^($pkgNameRE)-($versionRE)$/;

my $mainOutput = getMainOutput($prevBuild);

my $result =
{ storePath => $mainOutput->path
, id => $prevBuild->id
, version => $version
, outputName => $mainOutput->name
};
if (isValidPath($prevBuild->drvpath)) {
$result->{drvPath} = $prevBuild->drvpath;
}

return $result;
}

sub fetchInputSystemBuild {
my ($db, $project, $jobset, $name, $value) = @_;

my ($projectName, $jobsetName, $jobName, $attrs) = parseJobName($value);
$projectName ||= $project->name;
$jobsetName ||= $jobset->name;

my @latestBuilds = $db->resultset('LatestSucceededForJob')
->search({}, {bind => [$projectName, $jobsetName, $jobName]});

my @validBuilds = ();
foreach my $build (@latestBuilds) {
push(@validBuilds, $build) if isValidPath(getMainOutput($build)->path);
}
edit in src/lib/Hydra/Helper/AddBuilds.pm at line 26

[17.236]→[17.5332:5444](∅→∅),[17.5332]→[17.5332:5444](∅→∅),[17.5444]→[17.674:693](∅→∅),[17.693]→[17.5466:5472](∅→∅),[17.5466]→[17.5466:5472](∅→∅)

if (scalar(@validBuilds) == 0) {
print STDERR "input `", $name, "': no previous build available\n";
return ();
}
edit in src/lib/Hydra/Helper/AddBuilds.pm at line 27

[17.238]→[17.5477:5498](∅→∅),[17.5477]→[17.5477:5498](∅→∅),[17.5498]→[17.239:240](∅→∅),[17.240]→[17.5503:5654](∅→∅),[17.5503]→[17.5503:5654](∅→∅),[17.5654]→[17.241:242](∅→∅),[17.242]→[17.4546:4618](∅→∅),[17.714]→[17.4546:4618](∅→∅),[17.4618]→[17.5739:5810](∅→∅),[17.5739]→[17.5739:5810](∅→∅),[17.5810]→[17.243:244](∅→∅),[17.244]→[17.5827:5847](∅→∅),[17.5827]→[17.5827:5847](∅→∅),[17.5847]→[17.4890:4949](∅→∅),[17.364]→[17.5927:6091](∅→∅),[17.4949]→[17.5927:6091](∅→∅),[17.5927]→[17.5927:6091](∅→∅),[17.6091]→[17.245:266](∅→∅),[17.266]→[17.6137:6139](∅→∅),[17.6137]→[17.6137:6139](∅→∅),[17.6139]→[17.923:1014](∅→∅),[17.1014]→[17.0:1](∅→∅),[17.3441]→[17.0:1](∅→∅),[17.7419]→[17.0:1](∅→∅),[17.1]→[17.1015:2932](∅→∅),[17.2932]→[17.2204:2221](∅→∅),[17.2204]→[17.2204:2221](∅→∅),[17.2221]→[17.111:200](∅→∅),[17.89]→[17.0:16](∅→∅),[17.200]→[17.0:16](∅→∅),[17.764]→[17.0:16](∅→∅),[17.2281]→[17.0:16](∅→∅),[17.16]→[17.2281:2282](∅→∅),[17.622]→[17.2281:2282](∅→∅),[17.2281]→[17.2281:2282](∅→∅),[17.2282]→[17.765:867](∅→∅),[17.333]→[17.1515:1555](∅→∅),[17.867]→[17.1515:1555](∅→∅),[17.919]→[17.1515:1555](∅→∅),[17.8814]→[17.1515:1555](∅→∅),[17.2591]→[17.1515:1555](∅→∅),[17.1555]→[17.868:948](∅→∅),[17.948]→[17.2933:3042](∅→∅),[17.418]→[17.2591:2597](∅→∅),[17.948]→[17.2591:2597](∅→∅),[17.999]→[17.2591:2597](∅→∅),[17.1632]→[17.2591:2597](∅→∅),[17.3042]→[17.2591:2597](∅→∅),[17.8899]→[17.2591:2597](∅→∅),[17.2591]→[17.2591:2597](∅→∅),[17.2597]→[17.36:86](∅→∅),[17.86]→[17.10051:10086](∅→∅),[17.10051]→[17.10051:10086](∅→∅),[17.10086]→[17.949:988](∅→∅),[17.988]→[17.312:318](∅→∅),[17.781]→[17.312:318](∅→∅),[17.318]→[17.10146:10257](∅→∅),[17.2714]→[17.10146:10257](∅→∅),[17.10146]→[17.10146:10257](∅→∅),[17.10257]→[17.989:1028](∅→∅),[17.1028]→[17.319:325](∅→∅),[17.825]→[17.319:325](∅→∅),[17.325]→[17.10317:10328](∅→∅),[17.2725]→[17.10317:10328](∅→∅),[17.10317]→[17.10317:10328](∅→∅),[17.10328]→[17.1029:1095](∅→∅),[17.1095]→[3.0:84](∅→∅),[3.84]→[17.1160:1341](∅→∅),[17.1160]→[17.1160:1341](∅→∅),[17.1341]→[17.10391:10397](∅→∅),[17.2968]→[17.10391:10397](∅→∅),[17.10391]→[17.10391:10397](∅→∅),[17.10397]→[17.1377:1378](∅→∅),[17.1378]→[17.90:156](∅→∅),[17.156]→[17.201:257](∅→∅),[17.257]→[17.212:218](∅→∅),[17.212]→[17.212:218](∅→∅),[17.218]→[17.910:931](∅→∅),[17.1382]→[17.910:931](∅→∅),[17.910]→[17.910:931](∅→∅),[17.931]→[17.633:740](∅→∅),[17.740]→[17.2832:2973](∅→∅),[17.2973]→[17.863:876](∅→∅),[17.863]→[17.863:876](∅→∅),[17.876]→[17.2974:3000](∅→∅),[17.3000]→[17.899:925](∅→∅),[17.899]→[17.899:925](∅→∅),[17.925]→[17.1356:1358](∅→∅),[17.931]→[17.1356:1358](∅→∅),[17.1425]→[17.1356:1358](∅→∅),[17.10397]→[17.1356:1358](∅→∅),[17.1358]→[17.1383:1384](∅→∅),[17.1384]→[17.1358:1359](∅→∅),[17.1358]→[17.1358:1359](∅→∅),[17.1359]→[17.926:1032](∅→∅),[17.1032]→[17.3001:3570](∅→∅),[17.3570]→[17.1552:1565](∅→∅),[17.1552]→[17.1552:1565](∅→∅),[17.1565]→[17.3571:3652](∅→∅),[17.3652]→[7.0:57](∅→∅),[7.57]→[17.52:133](∅→∅),[17.52]→[17.52:133](∅→∅),[17.133]→[17.1385:1474](∅→∅),[17.3652]→[17.1385:1474](∅→∅),[17.1474]→[17.3652:4114](∅→∅),[17.3652]→[17.3652:4114](∅→∅),[17.4114]→[17.176:278](∅→∅),[17.278]→[8.0:108](∅→∅),[8.108]→[17.4114:4132](∅→∅),[17.367]→[17.4114:4132](∅→∅),[17.4114]→[17.4114:4132](∅→∅),[17.4132]→[17.2077:2105](∅→∅),[17.2077]→[17.2077:2105](∅→∅),[17.2105]→[17.4950:4951](∅→∅),[17.2105]→[17.1359:1379](∅→∅),[17.4951]→[17.1359:1379](∅→∅),[17.1359]→[17.1359:1379](∅→∅),[17.1379]→[17.2106:2143](∅→∅),[17.2143]→[17.1405:1424](∅→∅),[17.1405]→[17.1405:1424](∅→∅),[17.1424]→[10.0:50](∅→∅),[10.50]→[17.220:293](∅→∅),[17.1469]→[17.220:293](∅→∅),[17.293]→[17.361:412](∅→∅),[17.412]→[17.345:411](∅→∅),[17.345]→[17.345:411](∅→∅),[17.411]→[17.1469:1521](∅→∅),[17.1469]→[17.1469:1521](∅→∅),[17.1521]→[5.0:269](∅→∅),[5.269]→[17.213:257](∅→∅),[17.213]→[17.213:257](∅→∅),[17.257]→[5.270:452](∅→∅),[5.452]→[17.404:449](∅→∅),[17.404]→[17.404:449](∅→∅),[17.449]→[5.453:988](∅→∅),[17.851]→[17.2259:2295](∅→∅),[5.988]→[17.2259:2295](∅→∅),[17.2259]→[17.2259:2295](∅→∅),[17.2296]→[17.19:20](∅→∅),[17.450]→[17.450:451](∅→∅),[17.1379]→[17.2474:2489](∅→∅),[17.2474]→[17.2474:2489](∅→∅),[17.2489]→[17.2322:2392](∅→∅),[17.2392]→[17.2548:2609](∅→∅),[17.2548]→[17.2548:2609](∅→∅),[17.2609]→[14.83:155](∅→∅),[14.155]→[17.3249:3346](∅→∅),[17.3249]→[17.3249:3346](∅→∅),[17.3346]→[17.2779:2913](∅→∅),[17.2779]→[17.2779:2913](∅→∅),[17.2913]→[17.413:414](∅→∅),[17.414]→[17.2393:2483](∅→∅),[17.2528]→[17.2528:2529](∅→∅),[17.2529]→[17.0:379](∅→∅),[17.379]→[15.0:76](∅→∅),[15.76]→[17.2016:2107](∅→∅),[17.455]→[17.2016:2107](∅→∅),[17.3465]→[17.2016:2107](∅→∅),[17.2107]→[6.13:86](∅→∅),[6.86]→[17.2163:2180](∅→∅),[17.2163]→[17.2163:2180](∅→∅),[17.178]→[17.3181:3211](∅→∅),[17.2180]→[17.3181:3211](∅→∅),[17.3549]→[17.3181:3211](∅→∅),[17.3181]→[17.3181:3211](∅→∅),[17.3211]→[17.2181:2233](∅→∅),[17.1513]→[17.90:94](∅→∅),[17.94]→[17.0:137](∅→∅),[17.137]→[17.216:240](∅→∅),[17.216]→[17.216:240](∅→∅),[17.240]→[17.138:224](∅→∅),[17.224]→[17.423:478](∅→∅),[17.478]→[17.280:327](∅→∅),[17.280]→[17.280:327](∅→∅),[17.327]→[17.372:394](∅→∅),[17.372]→[17.372:394](∅→∅),[17.394]→[17.2316:2319](∅→∅),[17.1513]→[17.2316:2319](∅→∅),[17.2316]→[17.2316:2319](∅→∅),[17.2319]→[17.25:26](∅→∅),[17.26]→[17.3432:3507](∅→∅),[17.2319]→[17.3432:3507](∅→∅),[17.3432]→[17.3432:3507](∅→∅),[17.3507]→[17.1083:1198](∅→∅),[17.248]→[17.7099:7100](∅→∅),[17.1198]→[17.7099:7100](∅→∅),[17.186]→[17.7099:7100](∅→∅),[17.7100]→[17.2234:2292](∅→∅),[17.2292]→[17.7157:7193](∅→∅),[17.7157]→[17.7157:7193](∅→∅),[17.186]→[17.3600:3601](∅→∅),[17.494]→[17.3600:3601](∅→∅),[17.7193]→[17.3600:3601](∅→∅),[17.3600]→[17.3600:3601](∅→∅),[17.3601]→[17.7194:7402](∅→∅),[17.7402]→[17.2293:2362](∅→∅),[17.2362]→[17.7478:7575](∅→∅),[17.7478]→[17.7478:7575](∅→∅),[17.7575]→[17.3724:3725](∅→∅),[17.3724]→[17.3724:3725](∅→∅),[17.3860]→[17.3860:3898](∅→∅),[17.3898]→[17.1199:1272](∅→∅),[17.1272]→[17.4090:4091](∅→∅),[17.4090]→[17.4090:4091](∅→∅),[17.4186]→[17.4186:4677](∅→∅),[17.4677]→[17.7576:7803](∅→∅),[17.7803]→[17.495:528](∅→∅),[17.4892]→[17.495:528](∅→∅),[17.528]→[17.7804:7933](∅→∅),[17.7933]→[17.0:57](∅→∅),[17.528]→[17.0:57](∅→∅),[17.57]→[17.4133:4387](∅→∅),[17.4387]→[17.1273:1368](∅→∅),[17.1368]→[17.8023:8172](∅→∅),[17.8023]→[17.8023:8172](∅→∅),[17.391]→[17.663:701](∅→∅),[17.4550]→[17.663:701](∅→∅),[17.8172]→[17.663:701](∅→∅),[17.663]→[17.663:701](∅→∅),[17.701]→[17.0:93](∅→∅),[17.93]→[17.0:124](∅→∅),[17.793]→[17.0:124](∅→∅),[17.124]→[17.842:880](∅→∅),[17.1450]→[17.842:880](∅→∅),[17.842]→[17.842:880](∅→∅),[17.880]→[17.5165:5175](∅→∅),[17.5165]→[17.5165:5175](∅→∅),[17.5175]→[17.187:279](∅→∅),[17.279]→[17.1451:1523](∅→∅),[17.1523]→[17.345:374](∅→∅),[17.8247]→[17.345:374](∅→∅),[17.345]→[17.345:374](∅→∅),[17.374]→[17.94:168](∅→∅),[17.168]→[17.447:477](∅→∅),[17.447]→[17.447:477](∅→∅),[17.477]→[17.578:579](∅→∅),[17.579]→[17.411:438](∅→∅),[17.411]→[17.411:438](∅→∅),[17.438]→[17.580:581](∅→∅),[17.581]→[11.0:98](∅→∅),[11.98]→[17.9004:9045](∅→∅),[17.690]→[17.9004:9045](∅→∅),[17.690]→[17.5211:5250](∅→∅),[17.1383]→[17.5211:5250](∅→∅),[17.9045]→[17.5211:5250](∅→∅),[17.5211]→[17.5211:5250](∅→∅),[17.5250]→[17.691:724](∅→∅),[17.724]→[11.99:329](∅→∅),[11.329]→[17.261:359](∅→∅),[17.5581]→[17.261:359](∅→∅),[17.359]→[17.5581:5662](∅→∅),[17.5581]→[17.5581:5662](∅→∅),[17.5696]→[17.5696:5741](∅→∅),[17.5770]→[17.5770:5872](∅→∅),[17.5872]→[17.2419:2478](∅→∅),[17.2478]→[17.19661:19689](∅→∅),[17.4638]→[17.0:29](∅→∅),[17.29]→[12.80:131](∅→∅),[12.131]→[17.5872:5888](∅→∅),[17.1504]→[17.5872:5888](∅→∅),[17.4664]→[17.5872:5888](∅→∅),[17.5872]→[17.5872:5888](∅→∅),[17.5888]→[17.9046:9047](∅→∅),[17.9047]→[17.2479:2569](∅→∅),[17.2569]→[17.9144:9178](∅→∅),[17.9144]→[17.9144:9178](∅→∅),[17.9178]→[17.5888:5889](∅→∅),[17.5888]→[17.5888:5889](∅→∅),[17.5889]→[17.125:233](∅→∅),[17.233]→[17.1593:1667](∅→∅),[17.1593]→[17.1593:1667](∅→∅),[17.1667]→[17.725:726](∅→∅),[17.9255]→[17.725:726](∅→∅),[17.546]→[17.725:726](∅→∅),[17.726]→[17.19690:19799](∅→∅),[17.19799]→[17.6965:6996](∅→∅),[17.6965]→[17.6965:6996](∅→∅),[17.3068]→[17.459:460](∅→∅),[17.9374]→[17.459:460](∅→∅),[17.459]→[17.459:460](∅→∅),[17.9432]→[17.588:589](∅→∅),[17.588]→[17.588:589](∅→∅)

my @inputs = ();

foreach my $prevBuild (@validBuilds) {
my $pkgNameRE = "(?:(?:[A-Za-z0-9]|(?:-[^0-9]))+)";
my $versionRE = "(?:[A-Za-z0-9\.\-]+)";

my $relName = ($prevBuild->releasename or $prevBuild->nixname);
my $version = $2 if $relName =~ /^($pkgNameRE)-($versionRE)$/;

my $input =
{ storePath => getMainOutput($prevBuild)->path
, id => $prevBuild->id
, version => $version
, system => $prevBuild->system
};
push(@inputs, $input);
}

return @inputs;
}

sub fetchInputEval {
my ($db, $project, $jobset, $name, $value) = @_;

my $eval;

if ($value =~ /^\d+$/) {
$eval = $db->resultset('JobsetEvals')->find({ id => int($value) });
die "evaluation $eval->{id} does not exist\n" unless defined $eval;
} elsif ($value =~ /^($projectNameRE):($jobsetNameRE)$/) {
my $jobset = $db->resultset('Jobsets')->find({ project => $1, name => $2 });
die "jobset ‘$value’ does not exist\n" unless defined $jobset;
$eval = getLatestFinishedEval($jobset);
die "jobset ‘$value’ does not have a finished evaluation\n" unless defined $eval;
} elsif ($value =~ /^($projectNameRE):($jobsetNameRE):($jobNameRE)$/) {
$eval = $db->resultset('JobsetEvals')->find(
{ project => $1, jobset => $2, hasnewbuilds => 1 },
{ order_by => "id DESC", rows => 1
, where =>
\ [ # All builds in this jobset should be finished...
"not exists (select 1 from JobsetEvalMembers m join Builds b on m.build = b.id where m.eval = me.id and b.finished = 0) "
# ...and the specified build must have succeeded.
. "and exists (select 1 from JobsetEvalMembers m join Builds b on m.build = b.id where m.eval = me.id and b.job = ? and b.buildstatus = 0)"
, [ 'name', $3 ] ]
});
die "there is no successful build of ‘$value’ in a finished evaluation\n" unless defined $eval;
} else {
die;
}

my $jobs = {};
foreach my $build ($eval->builds) {
next unless $build->finished == 1 && $build->buildstatus == 0;
# FIXME: Handle multiple outputs.
my $out = $build->buildoutputs->find({ name => "out" });
next unless defined $out;
# FIXME: Should we fail if the path is not valid?
next unless isValidPath($out->path);
$jobs->{$build->get_column('job')} = $out->path;
}

return { jobs => $jobs };
}

sub fetchInput {
my ($plugins, $db, $project, $jobset, $name, $type, $value, $emailresponsible) = @_;
my @inputs;

if ($type eq "build") {
@inputs = fetchInputBuild($db, $project, $jobset, $name, $value);
}
elsif ($type eq "sysbuild") {
@inputs = fetchInputSystemBuild($db, $project, $jobset, $name, $value);
}
elsif ($type eq "eval") {
@inputs = fetchInputEval($db, $project, $jobset, $name, $value);
}
elsif ($type eq "string" || $type eq "nix") {
die unless defined $value;
@inputs = { value => $value };
}
elsif ($type eq "boolean") {
die unless defined $value && ($value eq "true" || $value eq "false");
@inputs = { value => $value };
}
else {
my $found = 0;
foreach my $plugin (@{$plugins}) {
@inputs = $plugin->fetchInput($type, $name, $value, $project, $jobset);
if (defined $inputs[0]) {
$found = 1;
last;
}
}
die "input `$name' has unknown type `$type'." unless $found;
}

foreach my $input (@inputs) {
$input->{type} = $type;
$input->{emailresponsible} = $emailresponsible;
}

return @inputs;
}

sub booleanToString {
my ($exprType, $value) = @_;
my $result;
if ($exprType eq "guile") {
if ($value eq "true") {
$result = "#t";
} else {
$result = "#f";
}
$result = $value;
} else {
$result = $value;
}
return $result;
}

sub buildInputToString {
my ($exprType, $input) = @_;
my $result;
if ($exprType eq "guile") {
$result = "'((file-name . \"" . ${input}->{storePath} . "\")" .
(defined $input->{revision} ? "(revision . \"" . $input->{revision} . "\")" : "") .
(defined $input->{revCount} ? "(revision-count . " . $input->{revCount} . ")" : "") .
(defined $input->{gitTag} ? "(git-tag . \"" . $input->{gitTag} . "\")" : "") .
(defined $input->{shortRev} ? "(short-revision . \"" . $input->{shortRev} . "\")" : "") .
(defined $input->{version} ? "(version . \"" . $input->{version} . "\")" : "") .
")";
} else {
$result = "{ outPath = builtins.storePath " . $input->{storePath} . "" .
"; inputType = \"" . $input->{type} . "\"" .
(defined $input->{uri} ? "; uri = \"" . $input->{uri} . "\"" : "") .
(defined $input->{revNumber} ? "; rev = " . $input->{revNumber} . "" : "") .
(defined $input->{revision} ? "; rev = \"" . $input->{revision} . "\"" : "") .
(defined $input->{revCount} ? "; revCount = " . $input->{revCount} . "" : "") .
(defined $input->{gitTag} ? "; gitTag = \"" . $input->{gitTag} . "\"" : "") .
(defined $input->{shortRev} ? "; shortRev = \"" . $input->{shortRev} . "\"" : "") .
(defined $input->{version} ? "; version = \"" . $input->{version} . "\"" : "") .
(defined $input->{outputName} ? "; outputName = \"" . $input->{outputName} . "\"" : "") .
(defined $input->{drvPath} ? "; drvPath = builtins.storePath " . $input->{drvPath} . "" : "") .
";}";
}
return $result;
}

sub inputsToArgs {
my ($inputInfo, $exprType) = @_;
my @res = ();

foreach my $input (sort keys %{$inputInfo}) {
push @res, "-I", "$input=$inputInfo->{$input}->[0]->{storePath}"
if scalar @{$inputInfo->{$input}} == 1
&& defined $inputInfo->{$input}->[0]->{storePath};
foreach my $alt (@{$inputInfo->{$input}}) {
if ($alt->{type} eq "string") {
push @res, "--argstr", $input, $alt->{value};
}
elsif ($alt->{type} eq "boolean") {
push @res, "--arg", $input, booleanToString($exprType, $alt->{value});
}
elsif ($alt->{type} eq "nix") {
die "input type ‘nix’ only supported for Nix-based jobsets\n" unless $exprType eq "nix";
push @res, "--arg", $input, $alt->{value};
}
elsif ($alt->{type} eq "eval") {
die "input type ‘eval’ only supported for Nix-based jobsets\n" unless $exprType eq "nix";
my $s = "{ ";
# FIXME: escape $_. But dots should not be escaped.
$s .= "$_ = builtins.storePath ${\$alt->{jobs}->{$_}}; "
foreach keys %{$alt->{jobs}};
$s .= "}";
push @res, "--arg", $input, $s;
}
else {
push @res, "--arg", $input, buildInputToString($exprType, $alt);
}
}
}

return @res;
}

sub evalJobs {
my ($inputInfo, $exprType, $nixExprInputName, $nixExprPath) = @_;

my $nixExprInput = $inputInfo->{$nixExprInputName}->[0]
or die "cannot find the input containing the job expression\n";
die "multiple alternatives for the input containing the Nix expression are not supported.\n"
if scalar @{$inputInfo->{$nixExprInputName}} != 1;
my $nixExprFullPath = $nixExprInput->{storePath} . "/" . $nixExprPath;

my $evaluator = ($exprType eq "guile") ? "hydra-eval-guile-jobs" : "hydra-eval-jobs";

my @cmd = ($evaluator, $nixExprFullPath, "--gc-roots-dir", getGCRootsDir, "-j", 1, inputsToArgs($inputInfo, $exprType));

if (defined $ENV{'HYDRA_DEBUG'}) {
sub escape {
my $s = $_;
$s =~ s/'/'\\''/g;
return "'" . $s . "'";
}
my @escaped = map escape, @cmd;
print STDERR "evaluator: @escaped\n";
}

(my $res, my $jobsJSON, my $stderr) = captureStdoutStderr(21600, @cmd);
die "$evaluator returned " . ($res & 127 ? "signal $res" : "exit code " . ($res >> 8))
. ":\n" . ($stderr ? decode("utf-8", $stderr) : "(no output)\n")
if $res;

print STDERR "$stderr";

return (decode_json($jobsJSON), $nixExprInput);
}

# Return the most recent evaluation of the given jobset (that
# optionally had new builds), or undefined if no such evaluation
# exists.
sub getPrevJobsetEval {
my ($db, $jobset, $hasNewBuilds) = @_;
my ($prevEval) = $jobset->jobsetevals(
($hasNewBuilds ? { hasnewbuilds => 1 } : { }),
{ order_by => "id DESC", rows => 1 });
return $prevEval;
}

# Check whether to add the build described by $buildInfo.
sub checkBuild {
my ($db, $jobset, $inputInfo, $nixExprInput, $buildInfo, $buildMap, $prevEval, $jobOutPathMap, $plugins) = @_;

my @outputNames = sort keys %{$buildInfo->{outputs}};
die unless scalar @outputNames;

# In various checks we can use an arbitrary output (the first)
# rather than all outputs, since if one output is the same, the
# others will be as well.
my $firstOutputName = $outputNames[0];
my $firstOutputPath = $buildInfo->{outputs}->{$firstOutputName};

my $jobName = $buildInfo->{jobName} or die;
my $drvPath = $buildInfo->{drvPath} or die;

my $build;

txn_do($db, sub {
my $job = $jobset->jobs->update_or_create({ name => $jobName });

# Don't add a build that has already been scheduled for this
# job, or has been built but is still a "current" build for
# this job. Note that this means that if the sources of a job
# are changed from A to B and then reverted to A, three builds
# will be performed (though the last one will probably use the
# cached result from the first). This ensures that the builds
# with the highest ID will always be the ones that we want in
# the channels. FIXME: Checking the output paths doesn't take
# meta-attributes into account. For instance, do we want a
# new build to be scheduled if the meta.maintainers field is
# changed?
if (defined $prevEval) {
# Only check one output: if it's the same, the other will be as well.
my $firstOutput = $outputNames[0];
my ($prevBuild) = $prevEval->builds->search(
# The "project" and "jobset" constraints are
# semantically unnecessary (because they're implied by
# the eval), but they give a factor 1000 speedup on
# the Nixpkgs jobset with PostgreSQL.
{ project => $jobset->project->name, jobset => $jobset->name, job => $jobName,
name => $firstOutputName, path => $firstOutputPath },
{ rows => 1, columns => ['id'], join => ['buildoutputs'] });
if (defined $prevBuild) {
#print STDERR " already scheduled/built as build ", $prevBuild->id, "\n";
$buildMap->{$prevBuild->id} = { id => $prevBuild->id, jobName => $jobName, new => 0, drvPath => $drvPath };
return;
}
}

# Prevent multiple builds with the same (job, outPath) from
# being added.
my $prev = $$jobOutPathMap{$jobName . "\t" . $firstOutputPath};
if (defined $prev) {
#print STDERR " already scheduled as build ", $prev, "\n";
return;
}

my $time = time();

sub null {
my ($s) = @_;
return $s eq "" ? undef : $s;
}

# Add the build to the database.
$build = $job->builds->create(
{ timestamp => $time
, description => null($buildInfo->{description})
, license => null($buildInfo->{license})
, homepage => null($buildInfo->{homepage})
, maintainers => null($buildInfo->{maintainers})
, maxsilent => $buildInfo->{maxSilent}
, timeout => $buildInfo->{timeout}
, nixname => $buildInfo->{nixName}
, drvpath => $drvPath
, system => $buildInfo->{system}
, nixexprinput => $jobset->nixexprinput
, nixexprpath => $jobset->nixexprpath
, priority => $buildInfo->{schedulingPriority}
, finished => 0
, iscurrent => 1
, ischannel => $buildInfo->{isChannel}
});

$build->buildoutputs->create({ name => $_, path => $buildInfo->{outputs}->{$_} })
foreach @outputNames;

$buildMap->{$build->id} = { id => $build->id, jobName => $jobName, new => 1, drvPath => $drvPath };
$$jobOutPathMap{$jobName . "\t" . $firstOutputPath} = $build->id;

print STDERR "added build ${\$build->id} (${\$jobset->project->name}:${\$jobset->name}:$jobName)\n";
});

return $build;
};
edit in src/lib/Hydra/Helper/Nix.pm at line 13

[13.122]

[17.127]

use IPC::Run;
replacement in src/script/hydra-eval-jobset at line 5

[17.151]→[17.473:492](∅→∅),[17.1363]→[17.473:492](∅→∅),[17.492]→[17.19854:19873](∅→∅),[17.19873]→[17.1746:1770](∅→∅),[17.492]→[17.1746:1770](∅→∅)

use Hydra::Schema;
use Hydra::Plugin;
use Hydra::Helper::Nix;

[17.151]

[17.10916]

use Config::General;
use Data::Dump qw(dump);
use Digest::SHA qw(sha256_hex);
use Encode;
use File::Slurp;
edit in src/script/hydra-eval-jobset at line 11

[17.10946]

[17.2430]

use Hydra::Helper::CatalystUtils;
edit in src/script/hydra-eval-jobset at line 13

[17.2456]

[17.688]

use Hydra::Helper::Nix;
replacement in src/script/hydra-eval-jobset at line 15

[17.710]→[17.2013:2045](∅→∅),[17.10946]→[17.2013:2045](∅→∅),[17.396]→[17.396:417](∅→∅),[17.417]→[17.2426:2451](∅→∅),[17.2451]→[17.2457:2472](∅→∅)

use Digest::SHA qw(sha256_hex);
use Config::General;
use Data::Dump qw(dump);
use Try::Tiny;

[17.710]

[17.0]

use Hydra::Plugin;
use Hydra::Schema;
use JSON;
edit in src/script/hydra-eval-jobset at line 19

[17.17]

[16.0]

use Nix::Store;
replacement in src/script/hydra-eval-jobset at line 21

[16.51]→[17.4394:4421](∅→∅),[17.67]→[17.4394:4421](∅→∅)

use JSON;
use File::Slurp;

[16.51]

[17.418]

use Try::Tiny;
edit in src/script/hydra-eval-jobset at line 33

[17.500]

[17.5400]

sub parseJobName {
# Parse a job specification of the form `<project>:<jobset>:<job>
# [attrs]'. The project, jobset and attrs may be omitted. The
# attrs have the form `name = "value"'.
my ($s) = @_;
our $key;
our %attrs = ();
# hm, maybe I should stop programming Perl before it's too late...
$s =~ / ^ (?: (?: ($projectNameRE) : )? ($jobsetNameRE) : )? ($jobNameRE) \s*
(\[ \s* (
([\w]+) (?{ $key = $^N; }) \s* = \s* \"
([\w\-]+) (?{ $attrs{$key} = $^N; }) \"
\s* )* \])? $
/x
or die "invalid job specifier `$s'";
return ($1, $2, $3, \%attrs);
}

sub attrsToSQL {
my ($attrs, $id) = @_;

my $query = "1 = 1";

foreach my $name (keys %{$attrs}) {
my $value = $attrs->{$name};
$name =~ /^[\w\-]+$/ or die;
$value =~ /^[\w\-]+$/ or die;
# !!! Yes, this is horribly injection-prone... (though
# name/value are filtered above). Should use SQL::Abstract,
# but it can't deal with subqueries. At least we should use
# placeholders.
$query .= " and exists (select 1 from buildinputs where build = $id and name = '$name' and value = '$value')";
}

return $query;
}

sub fetchInputBuild {
my ($db, $project, $jobset, $name, $value) = @_;

my $prevBuild;

if ($value =~ /^\d+$/) {
$prevBuild = $db->resultset('Builds')->find({ id => int($value) });
} else {
my ($projectName, $jobsetName, $jobName, $attrs) = parseJobName($value);
$projectName ||= $project->name;
$jobsetName ||= $jobset->name;

# Pick the most recent successful build of the specified job.
$prevBuild = $db->resultset('Builds')->search(
{ finished => 1, project => $projectName, jobset => $jobsetName
, job => $jobName, buildStatus => 0 },
{ order_by => "me.id DESC", rows => 1
, where => \ attrsToSQL($attrs, "me.id") })->single;
}

return () if !defined $prevBuild || !isValidPath(getMainOutput($prevBuild)->path);

#print STDERR "input `", $name, "': using build ", $prevBuild->id, "\n";

my $pkgNameRE = "(?:(?:[A-Za-z0-9]|(?:-[^0-9]))+)";
my $versionRE = "(?:[A-Za-z0-9\.\-]+)";

my $relName = ($prevBuild->releasename or $prevBuild->nixname);
my $version = $2 if $relName =~ /^($pkgNameRE)-($versionRE)$/;

my $mainOutput = getMainOutput($prevBuild);

my $result =
{ storePath => $mainOutput->path
, id => $prevBuild->id
, version => $version
, outputName => $mainOutput->name
};
if (isValidPath($prevBuild->drvpath)) {
$result->{drvPath} = $prevBuild->drvpath;
}

return $result;
}

sub fetchInputSystemBuild {
my ($db, $project, $jobset, $name, $value) = @_;

my ($projectName, $jobsetName, $jobName, $attrs) = parseJobName($value);
$projectName ||= $project->name;
$jobsetName ||= $jobset->name;

my @latestBuilds = $db->resultset('LatestSucceededForJob')
->search({}, {bind => [$projectName, $jobsetName, $jobName]});

my @validBuilds = ();
foreach my $build (@latestBuilds) {
push(@validBuilds, $build) if isValidPath(getMainOutput($build)->path);
}

if (scalar(@validBuilds) == 0) {
print STDERR "input `", $name, "': no previous build available\n";
return ();
}

my @inputs = ();

foreach my $prevBuild (@validBuilds) {
my $pkgNameRE = "(?:(?:[A-Za-z0-9]|(?:-[^0-9]))+)";
my $versionRE = "(?:[A-Za-z0-9\.\-]+)";

my $relName = ($prevBuild->releasename or $prevBuild->nixname);
my $version = $2 if $relName =~ /^($pkgNameRE)-($versionRE)$/;

my $input =
{ storePath => getMainOutput($prevBuild)->path
, id => $prevBuild->id
, version => $version
, system => $prevBuild->system
};
push(@inputs, $input);
}

return @inputs;
}

sub fetchInputEval {
my ($db, $project, $jobset, $name, $value) = @_;

my $eval;

if ($value =~ /^\d+$/) {
$eval = $db->resultset('JobsetEvals')->find({ id => int($value) });
die "evaluation $eval->{id} does not exist\n" unless defined $eval;
} elsif ($value =~ /^($projectNameRE):($jobsetNameRE)$/) {
my $jobset = $db->resultset('Jobsets')->find({ project => $1, name => $2 });
die "jobset ‘$value’ does not exist\n" unless defined $jobset;
$eval = getLatestFinishedEval($jobset);
die "jobset ‘$value’ does not have a finished evaluation\n" unless defined $eval;
} elsif ($value =~ /^($projectNameRE):($jobsetNameRE):($jobNameRE)$/) {
$eval = $db->resultset('JobsetEvals')->find(
{ project => $1, jobset => $2, hasnewbuilds => 1 },
{ order_by => "id DESC", rows => 1
, where =>
\ [ # All builds in this jobset should be finished...
"not exists (select 1 from JobsetEvalMembers m join Builds b on m.build = b.id where m.eval = me.id and b.finished = 0) "
# ...and the specified build must have succeeded.
. "and exists (select 1 from JobsetEvalMembers m join Builds b on m.build = b.id where m.eval = me.id and b.job = ? and b.buildstatus = 0)"
, [ 'name', $3 ] ]
});
die "there is no successful build of ‘$value’ in a finished evaluation\n" unless defined $eval;
} else {
die;
}

my $jobs = {};
foreach my $build ($eval->builds) {
next unless $build->finished == 1 && $build->buildstatus == 0;
# FIXME: Handle multiple outputs.
my $out = $build->buildoutputs->find({ name => "out" });
next unless defined $out;
# FIXME: Should we fail if the path is not valid?
next unless isValidPath($out->path);
$jobs->{$build->get_column('job')} = $out->path;
}

return { jobs => $jobs };
}

sub fetchInput {
my ($plugins, $db, $project, $jobset, $name, $type, $value, $emailresponsible) = @_;
my @inputs;

if ($type eq "build") {
@inputs = fetchInputBuild($db, $project, $jobset, $name, $value);
}
elsif ($type eq "sysbuild") {
@inputs = fetchInputSystemBuild($db, $project, $jobset, $name, $value);
}
elsif ($type eq "eval") {
@inputs = fetchInputEval($db, $project, $jobset, $name, $value);
}
elsif ($type eq "string" || $type eq "nix") {
die unless defined $value;
@inputs = { value => $value };
}
elsif ($type eq "boolean") {
die unless defined $value && ($value eq "true" || $value eq "false");
@inputs = { value => $value };
}
else {
my $found = 0;
foreach my $plugin (@{$plugins}) {
@inputs = $plugin->fetchInput($type, $name, $value, $project, $jobset);
if (defined $inputs[0]) {
$found = 1;
last;
}
}
die "input `$name' has unknown type `$type'." unless $found;
}

foreach my $input (@inputs) {
$input->{type} = $type;
$input->{emailresponsible} = $emailresponsible;
}

return @inputs;
}

sub booleanToString {
my ($exprType, $value) = @_;
my $result;
if ($exprType eq "guile") {
if ($value eq "true") {
$result = "#t";
} else {
$result = "#f";
}
$result = $value;
} else {
$result = $value;
}
return $result;
}

sub buildInputToString {
my ($exprType, $input) = @_;
my $result;
if ($exprType eq "guile") {
$result = "'((file-name . \"" . ${input}->{storePath} . "\")" .
(defined $input->{revision} ? "(revision . \"" . $input->{revision} . "\")" : "") .
(defined $input->{revCount} ? "(revision-count . " . $input->{revCount} . ")" : "") .
(defined $input->{gitTag} ? "(git-tag . \"" . $input->{gitTag} . "\")" : "") .
(defined $input->{shortRev} ? "(short-revision . \"" . $input->{shortRev} . "\")" : "") .
(defined $input->{version} ? "(version . \"" . $input->{version} . "\")" : "") .
")";
} else {
$result = "{ outPath = builtins.storePath " . $input->{storePath} . "" .
"; inputType = \"" . $input->{type} . "\"" .
(defined $input->{uri} ? "; uri = \"" . $input->{uri} . "\"" : "") .
(defined $input->{revNumber} ? "; rev = " . $input->{revNumber} . "" : "") .
(defined $input->{revision} ? "; rev = \"" . $input->{revision} . "\"" : "") .
(defined $input->{revCount} ? "; revCount = " . $input->{revCount} . "" : "") .
(defined $input->{gitTag} ? "; gitTag = \"" . $input->{gitTag} . "\"" : "") .
(defined $input->{shortRev} ? "; shortRev = \"" . $input->{shortRev} . "\"" : "") .
(defined $input->{version} ? "; version = \"" . $input->{version} . "\"" : "") .
(defined $input->{outputName} ? "; outputName = \"" . $input->{outputName} . "\"" : "") .
(defined $input->{drvPath} ? "; drvPath = builtins.storePath " . $input->{drvPath} . "" : "") .
";}";
}
return $result;
}

sub inputsToArgs {
my ($inputInfo, $exprType) = @_;
my @res = ();

foreach my $input (sort keys %{$inputInfo}) {
push @res, "-I", "$input=$inputInfo->{$input}->[0]->{storePath}"
if scalar @{$inputInfo->{$input}} == 1
&& defined $inputInfo->{$input}->[0]->{storePath};
foreach my $alt (@{$inputInfo->{$input}}) {
if ($alt->{type} eq "string") {
push @res, "--argstr", $input, $alt->{value};
}
elsif ($alt->{type} eq "boolean") {
push @res, "--arg", $input, booleanToString($exprType, $alt->{value});
}
elsif ($alt->{type} eq "nix") {
die "input type ‘nix’ only supported for Nix-based jobsets\n" unless $exprType eq "nix";
push @res, "--arg", $input, $alt->{value};
}
elsif ($alt->{type} eq "eval") {
die "input type ‘eval’ only supported for Nix-based jobsets\n" unless $exprType eq "nix";
my $s = "{ ";
# FIXME: escape $_. But dots should not be escaped.
$s .= "$_ = builtins.storePath ${\$alt->{jobs}->{$_}}; "
foreach keys %{$alt->{jobs}};
$s .= "}";
push @res, "--arg", $input, $s;
}
else {
push @res, "--arg", $input, buildInputToString($exprType, $alt);
}
}
}

return @res;
}
edit in src/script/hydra-eval-jobset at line 332

[17.5401]

[17.1424]

sub evalJobs {
my ($inputInfo, $exprType, $nixExprInputName, $nixExprPath) = @_;
edit in src/script/hydra-eval-jobset at line 335

[17.1425]

[17.483]

my $nixExprInput = $inputInfo->{$nixExprInputName}->[0]
or die "cannot find the input containing the job expression\n";
die "multiple alternatives for the input containing the Nix expression are not supported.\n"
if scalar @{$inputInfo->{$nixExprInputName}} != 1;
my $nixExprFullPath = $nixExprInput->{storePath} . "/" . $nixExprPath;

my $evaluator = ($exprType eq "guile") ? "hydra-eval-guile-jobs" : "hydra-eval-jobs";

my @cmd = ($evaluator, $nixExprFullPath, "--gc-roots-dir", getGCRootsDir, "-j", 1, inputsToArgs($inputInfo, $exprType));

if (defined $ENV{'HYDRA_DEBUG'}) {
sub escape {
my $s = $_;
$s =~ s/'/'\\''/g;
return "'" . $s . "'";
}
my @escaped = map escape, @cmd;
print STDERR "evaluator: @escaped\n";
}

(my $res, my $jobsJSON, my $stderr) = captureStdoutStderr(21600, @cmd);
die "$evaluator returned " . ($res & 127 ? "signal $res" : "exit code " . ($res >> 8))
. ":\n" . ($stderr ? decode("utf-8", $stderr) : "(no output)\n")
if $res;

print STDERR "$stderr";

return (decode_json($jobsJSON), $nixExprInput);
}

# Return the most recent evaluation of the given jobset (that
# optionally had new builds), or undefined if no such evaluation
# exists.
sub getPrevJobsetEval {
my ($db, $jobset, $hasNewBuilds) = @_;
my ($prevEval) = $jobset->jobsetevals(
($hasNewBuilds ? { hasnewbuilds => 1 } : { }),
{ order_by => "id DESC", rows => 1 });
return $prevEval;
}

# Check whether to add the build described by $buildInfo.
sub checkBuild {
my ($db, $jobset, $inputInfo, $nixExprInput, $buildInfo, $buildMap, $prevEval, $jobOutPathMap, $plugins) = @_;

my @outputNames = sort keys %{$buildInfo->{outputs}};
die unless scalar @outputNames;

# In various checks we can use an arbitrary output (the first)
# rather than all outputs, since if one output is the same, the
# others will be as well.
my $firstOutputName = $outputNames[0];
my $firstOutputPath = $buildInfo->{outputs}->{$firstOutputName};

my $jobName = $buildInfo->{jobName} or die;
my $drvPath = $buildInfo->{drvPath} or die;

my $build;

txn_do($db, sub {
my $job = $jobset->jobs->update_or_create({ name => $jobName });

# Don't add a build that has already been scheduled for this
# job, or has been built but is still a "current" build for
# this job. Note that this means that if the sources of a job
# are changed from A to B and then reverted to A, three builds
# will be performed (though the last one will probably use the
# cached result from the first). This ensures that the builds
# with the highest ID will always be the ones that we want in
# the channels. FIXME: Checking the output paths doesn't take
# meta-attributes into account. For instance, do we want a
# new build to be scheduled if the meta.maintainers field is
# changed?
if (defined $prevEval) {
# Only check one output: if it's the same, the other will be as well.
my $firstOutput = $outputNames[0];
my ($prevBuild) = $prevEval->builds->search(
# The "project" and "jobset" constraints are
# semantically unnecessary (because they're implied by
# the eval), but they give a factor 1000 speedup on
# the Nixpkgs jobset with PostgreSQL.
{ project => $jobset->project->name, jobset => $jobset->name, job => $jobName,
name => $firstOutputName, path => $firstOutputPath },
{ rows => 1, columns => ['id'], join => ['buildoutputs'] });
if (defined $prevBuild) {
#print STDERR " already scheduled/built as build ", $prevBuild->id, "\n";
$buildMap->{$prevBuild->id} = { id => $prevBuild->id, jobName => $jobName, new => 0, drvPath => $drvPath };
return;
}
}

# Prevent multiple builds with the same (job, outPath) from
# being added.
my $prev = $$jobOutPathMap{$jobName . "\t" . $firstOutputPath};
if (defined $prev) {
#print STDERR " already scheduled as build ", $prev, "\n";
return;
}

my $time = time();

sub null {
my ($s) = @_;
return $s eq "" ? undef : $s;
}

# Add the build to the database.
$build = $job->builds->create(
{ timestamp => $time
, description => null($buildInfo->{description})
, license => null($buildInfo->{license})
, homepage => null($buildInfo->{homepage})
, maintainers => null($buildInfo->{maintainers})
, maxsilent => $buildInfo->{maxSilent}
, timeout => $buildInfo->{timeout}
, nixname => $buildInfo->{nixName}
, drvpath => $drvPath
, system => $buildInfo->{system}
, nixexprinput => $jobset->nixexprinput
, nixexprpath => $jobset->nixexprpath
, priority => $buildInfo->{schedulingPriority}
, finished => 0
, iscurrent => 1
, ischannel => $buildInfo->{isChannel}
});

$build->buildoutputs->create({ name => $_, path => $buildInfo->{outputs}->{$_} })
foreach @outputNames;

$buildMap->{$build->id} = { id => $build->id, jobName => $jobName, new => 1, drvPath => $drvPath };
$$jobOutPathMap{$jobName . "\t" . $firstOutputPath} = $build->id;

print STDERR "added build ${\$build->id} (${\$jobset->project->name}:${\$jobset->name}:$jobName)\n";
});

return $build;
};
edit in tests/Setup.pm at line 7

[17.22]→[17.208:238](∅→∅),[17.208]→[17.208:238](∅→∅)

use Hydra::Helper::AddBuilds;
edit in tests/evaluation-tests.pl at line 4

[17.197]→[17.894:924](∅→∅),[17.894]→[17.894:924](∅→∅)

use Hydra::Helper::AddBuilds;