Evaluator cleanups

Dependencies

• [2] L3UFM7DK * $currentBuilds{id} can be 0.
• [3] R3ON2RJ3 hydra: missing argument
• [4] PIMGMGAF Rename hydra_eval_jobs to hydra-eval-jobs
• [5] 2WRTOU2Z Cleanup
• [6] SM5M2J3A Pass inputs to release expressions using -I
• [7] KJQWSRCC Speed up hydra-update-gc-roots
• [8] C5W3WZ3Y * Index the Builds table on drvPath to prevent a full table scan
• [9] KPZNJ33U * In views, support selecting a job that doesn't depend on the
• [10] NI5BVF2V * In job inputs of type "build", allow the project and jobset names of
• [11] S6OISBQ3 * Mark the "current" builds in a jobset, i.e. those corresponding to
• [12] RFE6T5LG * Store jobset evaluations in the database explicitly. This includes
• [13] SHZLOM5M * eval-jobs -> hydra_eval_jobs.
• [14] GA6XIXAN * Show the input bindings in evaluation error messages.
• [15] IXCUNELF * Don't bother with the Jobs.active column anymore.
• [16] 5SHCWE7X * Prevent repeated evaluation of a jobset with the same inputs. This
• [17] S5PV6IIM * Represent jobs explicitly in the DB.
• [18] D6YQQQCN * Don't ignore SIGCHLD after all, Perl doesn't like it. Just do
• [19] OOQ2D3KC * Refactoring: move fetchInput out of hydra_scheduler into a separate
• [20] X27GNHDV * Basic job info in the database.
• [21] H7CNGK4O * Log evaluation errors etc. in the DB.
• [22] EDRUQ4UK Die TABs die
• [23] HPEG2RHV Merge the BuildResultInfo table into the Builds table
• [24] IWHFLFVV * Randomly permute the order in which builds are added. This is
• [25] GCHNNFZP
• [26] 3XTHEUMP * Implemented the clone feature.
• [27] 4IXVBLUI hack to try and prevent too many newlines
• [28] 2I2ZX6JB * Make the "latest succeeded" query (used by the "latest" channel)
• [29] A22P7HCO hydra: at evaluation, check if path is already built, and mark as built in stead of adding to the queue.
• [30] FM4O2L4M hydra: if evaluator sees cached build, also add the buildproducts
• [31] JZE7DC2F Whitespace
• [32] MOX7XJ2E Merge the BuildSchedulingInfo table into the Builds table
• [33] ZDENAYQI * email notification of evaluation errors to project owner (if desired)
• [34] O25D52TA initial support for mercurial
• [35] FDE3BJAP * Refactoring.
• [36] LZO3C2KI * Hack around those SQLite timeouts: just retry the transaction.
• [37] CVDK3XJK * In the scheduler, don't check if we've already done a build (except
• [38] JM3DPYOM generated schema with new dbix class schema loader, grrrrrr
• [39] PT5XCF2N Add another index to speed up the jobstatus page
• [40] 4D4U5IPY * Allow jobsets to be disabled.
• [41] SQQSV4NB * Top-level: don't exit on errors.
• [42] TIOBBINA * Some renaming.
• [43] POPU2ATH * hydra_scheduler: use eval-jobs.
• [44] TULPZ62Y * Perform builds in parallel.
• [45] M552HLIA * Support variant builds.
• [46] QE253KG4 * Option to show the tail of a log.
• [47] WRIU3S5E * UI for cloning builds (not functional yet).
• [*] LBNVQXUB * Build the /build stuff in a separate controller.
• [*] BMSQD2ZH Indentation
• [*] N22GPKYT * Put info about logs / build products in the DB.
• [*] T2232OBS * Add some DB indices to make the /releases page much faster.
• [*] 6QRHXIM3 * Speed up the jobset index page. Especially the query to get the
• [*] 34DPX2OR * Create some more indices on tables that have a foreign key reference
• [*] D5QIOJGP * Move everything up one directory.

Change contents

• replacement in src/lib/Hydra/Controller/Build.pm at line 541
  [8.1031]→[8.1031:1090](∅→∅)
  − $inputInfo, $nixExprInput, $job, \%currentBuilds);
  [8.1031]

  [8.1090]
  + $inputInfo, $nixExprInput, $job, \%currentBuilds, undef);
• edit in src/lib/Hydra/Helper/AddBuilds.pm at line 20
  [50.156]

  [50.156]
  + getPrevJobsetEval
• edit in src/lib/Hydra/Helper/AddBuilds.pm at line 788
  [50.1513]

  [8.2316]
  + }

  +

  +

  + # Return the most recent evaluation of the given jobset that had new

  + # builds, or undefined if no such evaluation exists.

  + sub getPrevJobsetEval {

  + my ($db, $jobset) = @_;

  + my ($prevEval) = $jobset->jobsetevals({ hasnewbuilds => 1 }, { order_by => "id DESC", rows => 1 });

  + return $prevEval;
• replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 802
  [8.3507]→[8.3507:3600](∅→∅)
  − my ($db, $project, $jobset, $inputInfo, $nixExprInput, $buildInfo, $currentBuilds) = @_;
  [8.3507]

  [8.3600]
  + my ($db, $project, $jobset, $inputInfo, $nixExprInput, $buildInfo, $buildIds, $prevEval) = @_;
• replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 835
  [8.4892]→[8.4892:5079](∅→∅),[8.5079]→[8.641:707](∅→∅),[8.707]→[8.5145:5165](∅→∅),[8.5145]→[8.5145:5165](∅→∅)
  − my @previousBuilds = $job->builds->search({outPath => $outPath, isCurrent => 1});

  − if (scalar(@previousBuilds) > 0) {

  − print STDERR "already scheduled/built\n";

  − $currentBuilds->{$_->id} = 0 foreach @previousBuilds;

  − return;
  [8.4892]

  [8.5165]
  + if (defined $prevEval) {

  + my ($prevBuild) = $prevEval->builds->search({ job => $job->name, outPath => $outPath }, { rows => 1, columns => ['id'] });

  + if (defined $prevBuild) {

  + print STDERR " already scheduled/built as build ", $prevBuild->id, "\n";

  + $buildIds->{$prevBuild->id} = 0;

  + return;

  + }
• edit in src/lib/Hydra/Helper/AddBuilds.pm at line 847
  [8.5211]

  [8.5211]
  + my %extraFlags;

  + if (isValidPath($outPath)) {

  + %extraFlags =

  + ( finished => 1

  + , iscachedbuild => 1

  + , buildstatus => 0

  + , starttime => $time

  + , stoptime => $time

  + , logfile => getBuildLog($drvPath)

  + , errormsg => ""

  + , releasename => getReleaseName($outPath)

  + );

  + } else {

  + %extraFlags = ( finished => 0 );

  + }

  +
• replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 864
  [8.5250]→[8.5250:5278](∅→∅),[8.5278]→[8.439:473](∅→∅)
  − { finished => 0

  − , timestamp => $time
  [8.5250]

  [8.5312]
  + { timestamp => $time
• edit in src/lib/Hydra/Helper/AddBuilds.pm at line 879
  [8.5872]

  [8.5872]
  + , priority => $priority

  + , busy => 0

  + , locker => ""

  + , %extraFlags
• replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 885
  [8.5965]→[8.5965:6007](∅→∅)
  − $currentBuilds->{$build->id} = 1;
  [8.5889]

  [8.6007]
  + $buildIds->{$build->id} = 1;
• replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 887
  [8.6016]→[5.1426:1463](∅→∅),[5.1463]→[8.510:580](∅→∅),[8.510]→[8.510:580](∅→∅),[8.580]→[8.4619:4710](∅→∅),[8.4710]→[8.715:987](∅→∅),[8.715]→[8.715:987](∅→∅)
  − if (isValidPath($outPath)) {

  − print STDERR "marked as cached build ", $build->id, "\n";

  − $build->update(

  − { finished => 1

  − , iscachedbuild => 1

  − , buildstatus => 0

  − , starttime => $time

  − , stoptime => $time

  − , logfile => getBuildLog($drvPath)

  − , errormsg => ""

  − , releasename => getReleaseName($outPath)

  − });
  [8.6016]

  [3.0]
  + if ($build->iscachedbuild) {

  + print STDERR " marked as cached build ", $build->id, "\n";
• replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 891
  [8.1004]→[8.1004:1075](∅→∅),[8.1075]→[8.2942:2989](∅→∅),[8.2989]→[8.1173:1252](∅→∅),[8.1173]→[8.1173:1252](∅→∅)
  − print STDERR "added to queue as build ", $build->id, "\n";

  − $build->update(

  − { priority => $priority

  − , busy => 0

  − , locker => ""

  − });
  [8.1004]

  [8.1252]
  + print STDERR " added to queue as build ", $build->id, "\n";
• edit in src/lib/Hydra/Schema/JobsetEvals.pm at line 164
  [8.1278]→[8.4789:4879](∅→∅)
  − # You can replace this text with custom content, and it will be preserved on regeneration
• replacement in src/script/hydra-evaluator at line 83
  [8.1440]→[8.1440:1496](∅→∅)
  − print $email->as_string if $ENV{'HYDRA_MAIL_TEST'};
  [8.1440]

  [8.1496]
  + print STDERR $email->as_string if $ENV{'HYDRA_MAIL_TEST'};
• edit in src/script/hydra-evaluator at line 87
  [8.1521]

  [8.1521]
  +
• replacement in src/script/hydra-evaluator at line 109
  [4.295]→[8.2104:2174](∅→∅),[8.2104]→[8.2104:2174](∅→∅)
  − # JobsetInputHashes to see if we've already evaluated this set of
  [4.295]

  [8.2174]
  + # JobsetInputHashes to see if the previous evaluation had the same
• replacement in src/script/hydra-evaluator at line 113
  [8.2334]→[8.2334:2335](∅→∅),[8.2335]→[6.412:485](∅→∅),[6.485]→[8.2400:2449](∅→∅),[8.5522]→[8.2400:2449](∅→∅),[8.2400]→[8.2400:2449](∅→∅)
  −

  − if (scalar($jobset->jobsetevals->search({hash => $argsHash})) > 0) {

  − print " already evaluated, skipping\n";
  [8.2334]

  [8.2449]
  + my $prevEval = getPrevJobsetEval($db, $jobset);

  + if ($prevEval->hash eq $argsHash) {

  + print STDERR " jobset is unchanged, skipping\n";
• edit in src/script/hydra-evaluator at line 129
  [8.5278]

  [8.5576]
  + # Clear the "current" flag on all builds. Since we're in a

  + # transaction this will only become visible after the new

  + # current builds have been added.

  + $jobset->builds->search({iscurrent => 1})->update({iscurrent => 0});

  +
• replacement in src/script/hydra-evaluator at line 135
  [8.5628]→[8.5628:5655](∅→∅)
  − my %currentBuilds;
  [8.5628]

  [8.5655]
  + my %buildIds;
• replacement in src/script/hydra-evaluator at line 138
  [8.5750]→[8.5750:5911](∅→∅)
  − print "considering job " . $job->{jobName} . "\n";

  − checkBuild($db, $project, $jobset, $inputInfo, $nixExprInput, $job, \%currentBuilds);
  [8.5750]

  [8.5911]
  + print STDERR " considering job " . $project->name, ":", $jobset->name, ":", $job->{jobName} . "\n";

  + checkBuild($db, $project, $jobset, $inputInfo, $nixExprInput, $job, \%buildIds, $prevEval);
• replacement in src/script/hydra-evaluator at line 149
  [8.2042]→[8.1183:1364](∅→∅),[8.1364]→[8.6513:6534](∅→∅),[8.6513]→[8.6513:6534](∅→∅),[8.6534]→[8.1365:1416](∅→∅),[8.1416]→[8.6589:6613](∅→∅),[8.6589]→[8.6589:6613](∅→∅)
  − foreach my $job ($jobset->jobs->all) {

  − if ($failedJobNames{$job->name}) {

  − $job->update({errormsg => join '\n', @{$failedJobNames{$job->name}}});

  − } else {

  − $job->update({errormsg => undef});

  − }

  − }
  [8.2042]

  [8.5548]
  + $_->update({ errormsg => $failedJobNames{$_->name} ? join '\n', @{$failedJobNames{$_->name}} : undef })

  + foreach $jobset->jobs->all;
• edit in src/script/hydra-evaluator at line 152
  [8.5549]→[8.5549:5708](∅→∅),[8.5708]→[2.0:88](∅→∅),[2.88]→[8.5923:5934](∅→∅),[8.5839]→[8.5923:5934](∅→∅)
  − # Clear the "current" flag on all builds that are no longer

  − # current.

  − foreach my $build ($jobset->builds->search({iscurrent => 1})) {

  − $build->update({iscurrent => 0}) unless defined $currentBuilds{$build->id};

  − }

  −
• replacement in src/script/hydra-evaluator at line 153
  [8.5964]→[8.5964:6019](∅→∅)
  − while (my ($id, $new) = each %currentBuilds) {
  [8.5964]

  [8.6019]
  + while (my ($id, $new) = each %buildIds) {
• replacement in src/script/hydra-evaluator at line 166
  [8.6380]→[8.6380:6439](∅→∅)
  − while (my ($id, $new) = each %currentBuilds) {
  [8.6380]

  [8.6439]
  + while (my ($id, $new) = each %buildIds) {
• replacement in src/script/hydra-evaluator at line 172
  [8.5866]→[8.689:755](∅→∅),[8.6630]→[8.689:755](∅→∅),[8.689]→[8.689:755](∅→∅)
  − # Store the errors messages for jobs that failed to evaluate.
  [8.5866]

  [8.755]
  + # Store the error messages for jobs that failed to evaluate.
• replacement in src/script/hydra-evaluator at line 195
  [8.2106]→[8.2106:2184](∅→∅)
  − print "considering jobset ", $jobset->name, " in ", $project->name, "\n";
  [8.2106]

  [8.2184]
  + print STDERR "considering jobset ", $project->name, ":", $jobset->name, "\n";
• replacement in src/script/hydra-evaluator at line 203
  [8.2288]→[8.2288:2355](∅→∅)
  − print "error evaluating jobset ", $jobset->name, ": $msg";
  [8.2288]

  [8.1624]
  + print STDERR "error evaluating jobset ", $jobset->name, ": $msg";
• replacement in src/script/hydra-evaluator at line 214
  [8.1831]→[8.4407:4467](∅→∅)
  − print "considering project ", $project->name, "\n";
  [8.1831]

  [8.3184]
  + print STDERR "considering project ", $project->name, "\n";
• replacement in src/script/hydra-evaluator at line 235
  [8.37]→[8.37:65](∅→∅),[8.65]→[8.995:1022](∅→∅),[8.995]→[8.995:1022](∅→∅)
  − if ($@) { print "$@"; }

  − print "sleeping...\n";
  [8.37]

  [8.855]
  + if ($@) { print STDERR "$@"; }

  + print STDERR "sleeping...\n";
• edit in src/sql/hydra.sql at line 496
  [52.2228]

  [53.2113]
  +
• edit in src/sql/hydra.sql at line 521
  [8.54]→[7.3042:3274](∅→∅)
  − create index IndexBuildsOnKeep on Builds(keep); -- used by hydra-update-gc-roots

  − create index IndexMostRecentSuccessfulBuilds on Builds(project, jobset, job, system, finished, buildStatus, id desc); -- used by hydra-update-gc-roots
• edit in src/sql/hydra.sql at line 530
  [54.139]
  +

  + -- For hydra-update-gc-roots.

  + create index IndexBuildsOnKeep on Builds(keep);

  + create index IndexMostRecentSuccessfulBuilds on Builds(project, jobset, job, system, finished, buildStatus, id desc);

  +

  + -- To get the most recent eval for a jobset.

  + create index IndexJobsetEvalsOnJobsetId on JobsetEvals(project, jobset, hasNewBuilds, id desc);
• file addition: upgrade-4.sql (----------)
  [55.3004]
  + create index IndexJobsetEvalsOnJobsetId on JobsetEvals(project, jobset, hasNewBuilds, id desc);