Remove hydra-build and the old hydra-queue-runner

Dependencies

• [2] MREXL5RT Whoops
• [3] Z4Y3TVEE hydra-queue-{runner,evaluator}: don't clutter the system log with debug messages
• [4] ZNGKTEAI If a build aborts, mark any remaining active build steps as aborted
• [5] 6SGBF7JP Set build status to 1 if the primary build failed
• [6] WMSCRPSC Fill in starttime/stoptime for cached builds
• [7] 6PG5BATL Remove tabs
• [8] PHNLYPKB Call buildFinished when a cached build is added
• [9] VXTPW6BF hydra-build: Hack to handle timeouts
• [10] FUJ3A66L hydra-queue-runner: Set the start time properly
• [11] MMUBEIGV hydra-queue-runner: Tweaked the selection method
• [12] IEP2Q6GV *headdesk*
• [13] TSR3RZC4 Revert "Add a dependency_lookup configuration option to enable (slow) dependency lookup in queue. This behaviour was disabled temporarily in accefbb79 due to slowness in very large queues, but some people might be dependent on it, so it is configurable until the previous behaviour is implemented more efficiently."
• [14] AWMM5OGV Use delete instead of delete_all
• [15] ZZYTUBG2 Fix extreme slowness in hydra-queue-runner
• [16] 533PIYC3 Prevent a division by zero in hydra-queue-runner
• [17] IS7GUIWY hydra-eval-guile-jobs: Register derivations as GC roots.
• [18] COLZ7V4E hydra-build: Handle new trace messages
• [19] 3YHNO5H2 Don't use Perl's -w flag
• [20] UJOSXBZI Record which build a failed build step was propagated from
• [21] E746D2DR Don't show missing paths in logs
• [22] WJ7IKFXL Fix not-null constraint violation inserting build step
• [23] QMC6IMJQ Split timeSpent query into 2 separate queries, as postgresql isn't able to figure out a decent query plan. With 120k jobs in queue, this makes some queries go from 100s to 1-2s.
• [24] 6MGFQDR2 deleted some old scripts
• [25] QMW24O5S Add support for Guile & Guix.
• [26] NREF6YOA * Don't start more builds concurrently than allowed for each system
• [27] FV2M6MOT hydra: use autoconf/-make
• [28] DBPIYHMA hydra: add nix-prefetch-* to tarball
• [29] QI7MGZAU
• [30] K3EAQY3X * Doh.
• [31] JR76VU3W Remove timeout detection hack
• [32] A22P7HCO hydra: at evaluation, check if path is already built, and mark as built in stead of adding to the queue.
• [33] JK2QWPH6
• [34] 4LR7CSBT hydra-build: Give a nicer error message if the derivation is gone
• [35] D5QIOJGP * Move everything up one directory.
• [36] 4N5APGRG * Start of a helper tool to evaluate job expressions efficiently.
• [37] KX5L74EY add nix-prefetch- scripts for now, were externals in svn
• [38] TRFRGOBD Remove Twitter notification support
• [39] TFLAR4KA hydra-queue-runner: Start as many builds as possible on each iteration
• [40] CURCK6C2
• [41] 3PNG7NIB Remove trailing whitespace
• [42] CQJX3RGU hydra-queue-runner: Don't kill builds we just started
• [43] EUWLW7FY Don't use the Switch module
• [44] IW2LHCLL fixed email bug
• [45] B72GLND4
• [46] CMU3YKOU * Store the release name.
• [47] 4SJPAAJX Pass failing dependent builds to buildFinished
• [48] M2CFFNJY Remove unused file
• [49] 37R34XJO * Negative caching: don't perform a build if a dependency already
• [50] Q6SOGMDQ Hydra/28: Rename "scheduler" to "evaluator"
• [51] VI32YSGA I should test first
• [52] 2KDHXY52 Kill builds that produce more than 64 MiB of log output
• [53] 2PWOXJTX
• [54] XYBSTHMJ fix for buildsteps starting at 0, probably something changed in catalyst
• [55] BOFOHCPK removed debug print, added last 50 lines in failure emails
• [56] FHF6IZJQ * Basic release management: releases are now dynamically computed as
• [57] A32UT34U hydra_build.pl: Honor `$build->timeout'.
• [58] KBW3FDZ2 Merge remote branch 'remotes/origin/master'
• [59] TJK27WSB Open the DB using Hydra::Model::DB->new
• [60] 6L3ZM55S Add font for the captcha
• [61] NB2VOKIR Include names of committers in HipChat notifications
• [62] X27GNHDV * Basic job info in the database.
• [63] MPGVCHVF * Fix an apparent incompatibility with recent DBIx::Class.
• [64] CXRCPDSQ * added support for twitter notification
• [65] I7UELKBV hydra-queue-runner: Don't unlock builds we just started
• [66] OG7BEM57
• [67] UVNQPK3T Hydra/56: handle failed builds with result only at build level, not buildsteps
• [68] S3ZLZP3N * Cut off builds after half an hour of apparent inactivity. This
• [69] D7NXMCON * Doh.
• [70] A63IHCMX * Register GC roots properly.
• [71] UMOJJ6DV time out to 3600 for now
• [72] PKPWUHUX * Idem.
• [73] LSPRR4TK hydra-queue-runner: Disable findBuildDependencyInQueue for now
• [74] D7TT2BNK
• [75] J5UVLXOK * Start of a basic Catalyst web interface.
• [76] 6KCP6ODP * Get the URI for use in notification mails from the Hydra config
• [77] VJP6O6WA * Doh. Remove debug statement.
• [78] AS5PAYLI
• [79] GAIBDEZZ * Store the name of the machine that performed a build step in the
• [80] 6US6LEC7 * Add a NarSize field to Hydra manifests. This allows nix-env
• [81] QZLMDKMU * Queue runner: don't start scheduled builds builds if they belong to
• [82] FHAVPTZ6 Hydra/23: added some X-headers with meta info in email notifications, added more descriptive status
• [83] OV7F5M3E Merge branch 'queue-17'
• [84] YTESD75Z * Get rid of zombies.
• [85] NLJJZVHO * Use ->update({...}) properly.
• [86] R6B5CAFF Let Builds.timestamp refer to the time the build was added
• [87] WTUBAA5Q hydra-queue-runner: Handle restarted builds whose derivation is gone
• [88] US27ZTX5 * HydraFrontend -> Hydra.
• [89] EFWN7JBV * Added a status page that shows all the currently executing build steps.
• [90] CLXEECMF * Start putting build results in a database.
• [91] G5A7TZVI * Don't discard old build steps when restarting a build.
• [92] 7YBYT2LQ
• [93] DGZE7ZRA * Don't log redundant build steps in case of cached failures.
• [94] RWIBJ5L4 * Autoflush stdout.
• [95] FYO6NECE hydra
• [96] AMFMXR52 Provide a command ‘hydra-init’ to initialise/upgrade the database
• [97] RBHHV7P7 * Read logs using logContents function in stead of handling it everywhere separately.
• [98] UGA45FNC Add a plugin for backing up builds in s3
• [99] SX5XYD6I When propagating failure, propagate the duration and machine
• [100] 24BMQDZA Start of single-process hydra-queue-runner
• [101] DQD7JMSU * Fix the terminology.
• [102] R5D7DZPE
• [103] 5WCR3MKU trunk -> master
• [104] 3BKF6P72 * Use Nix's negative caching.
• [105] 3E6IP3R3 * Add the name of the jobset to ReleaseSetJobs, otherwise we can't
• [106] WOHG5MJD hydra: catalyst plain view doesn;t work with data => 0, so put some whitespace around it
• [107] Y6AHH4TH Remove the logfile and logSize columns from the database
• [108] RNNLIVYY Respect SystemTypes if defined
• [109] JGLE5BRN Add separate build step status codes for cached failures and timeouts
• [110] GEADFVZ5 hydra-queue-runner: Improved scheduling
• [111] MNZ67UXO If a build step fail, immediately fail all queued builds that depend on it
• [112] X3YLTWJA hydra-queue-runner: Cache the lookup of time spent per jobset
• [113] N22GPKYT * Put info about logs / build products in the DB.
• [114] TWVSALRL * Allow the maximum number of concurrent builds per platform to be
• [115] BLVQGJ4L Use OO-style plugins
• [116] XHOZT4WT Add a command `hydra-create-user' for managing user accounts
• [117] GZAXDYBQ * Script for initialising the database.
• [118] 7DWCXNC7 Use the new Nix Perl bindings
• [119] YEXD7CBK Fix findBuildDependencyInQueue
• [120] UCVYVMXB Use the same start/stop time for the build steps as for the build
• [121] QAJK5MCE Remove obsolete hydra-control script
• [122] D6YQQQCN * Don't ignore SIGCHLD after all, Perl doesn't like it. Just do
• [123] L4AI5YL6 Rename hydra_*.pl to hydra-*
• [124] YTIDBFGU Drop unused "disabled" columns
• [125] JUTAEI7R Restore old findBuildDependencyInQueue behaviour
• [126] YFPZ46YK * hydra: added variant of build input type, 'build output (same system)' to allow better continous integration in one jobset for multiple system. it makes sure that the system of the build that is passed as input for a job has the same system as the job.
• [127] GYPHTT4M Manual: Remove tabs, indent consistently
• [128] EX4FXA5T Handle active build steps of aborted builds properly
• [129] FLPZ3YCK Remove debug line
• [130] IUCHXUJP Use "can" to check whether a plugin supports buildFinished
• [131] D3DIBMOK * For products that are directories (like manuals), allow a default
• [132] A5V7HLIQ * Shut up DBIx::Class.
• [133] 2T42QGZD * Register builds as GC roots so they don't get deleted.
• [134] HE3GX5IP Optimize fetch-git.
• [135] AFTXA575 * $HYDRA_DATA environment variable.
• [136] T4LLYESZ * Nix expression for building Hydra.
• [137] EKNK5AHQ doc: Augment the "Installation" section with material by Visser & Dolstra.
• [138] J4AJQ6RX Add a dependency_lookup configuration option to enable (slow) dependency lookup in queue. This behaviour was disabled temporarily in accefbb79 due to slowness in very large queues, but some people might be dependent on it, so it is configurable until the previous behaviour is implemented more efficiently.
• [139] 7VWDMKAZ hydra-queue-runner: don't clutter the system log with debug message
• [140] HPEG2RHV Merge the BuildResultInfo table into the Builds table
• [141] WHAFVCEI
• [142] 64K7R4Y6 Forgot to change Nix to Nix::Store in one place
• [143] TPSCSZKX Speed up findBuildDependencyInQueue
• [144] AZ4LR2GT * Scripts for starting / stopping Hydra.
• [145] IJPTEKRF
• [146] QUTWJR7P * Include more info in notification emails.
• [147] ULHEOJO2 * Better timeout.
• [148] DN3VAAP4 hydra: remove HYDRA_HOME env var, not used
• [149] 2IEFMER5 * Add --fallback to prevent problems with obsolete substitutes.
• [150] TULPZ62Y * Perform builds in parallel.
• [151] BD3GRK4B * Get rid of "positive failures" and separate log phases.
• [152] WZ3AEJ67 * hydra_update_gc_roots.pl registers build outputs that should be kept
• [153] ARD6Z67T Do incremental SVN checkouts
• [154] CLJQCY2X * Store info about all the build actions and allow them to be
• [155] 5EQYVRWE Add a plugin mechanism
• [156] MOX7XJ2E Merge the BuildSchedulingInfo table into the Builds table
• [157] G3IUM7VL hydra: add support for succeedOnFailure feature in stdenv
• [158] 5DSF5KWY * Perl sucks. "getBuildLog $drvPath" doesn't mean the same as
• [159] 57Y7RG2A Update queryPathInfo calls
• [160] L2E6EVE2 * Merged the Build and Job tables.
• [161] WQXF2T3D hydra-evaluator: Don't require $HYDRA_CONFIG
• [162] UN2KZL3A Rename c -> hydra-eval-jobs
• [163] PYTQMQKD Hydra/17: in queue runner, prefer builds in the queue that are a dependency of another build (with higher priority)
• [164] WM5WAST3 * Monitor for dead builds all the time.
• [165] ZTQEU5QS Hydra: Add support for maxSilent meta attribute (also already added timeout, but not implemented the actual timeout for the build yet)
• [166] ERNOO5ZZ * Reorganising.
• [167] LZO3C2KI * Hack around those SQLite timeouts: just retry the transaction.
• [168] 5O6E5SU5 hydra: store logfile/output path/closure size
• [169] LQNBKF3D
• [170] AHTEIK7G * Added a maintainers field to the Builds table.
• [171] KA45EBF5 * Send email if a build fails.
• [172] W5GW4E22 Don't install hydra-create
• [173] Y6H7Y3OT Capture the path to `guile', when available.
• [174] ECBA3GQO * Make the schema class names match the case of the SQL table names.
• [175] PMNWRTGJ Add multiple output support
• [176] SA5ZZ3I4 hydra-queue-runner: Use nix.machines instead of the SystemTypes table to determine how many build jobs are allowed per system type.
• [177] DTXTS7LN * Speed up findBuildDependencyInQueue by doing only one SQL query for
• [178] 66MEE6QG hydra-build: Don't send a giant query to the database
• [179] SIZ6KXPW Set the build status properly for failing local builds
• [180] T7CCJQOF * Revert for now due to Postgres breakage.

Change contents

• replacement in doc/manual/installation.xml at line 103
  [24.3391]→[24.3391:3473](∅→∅),[24.3035]→[24.1398:1399](∅→∅),[24.3473]→[24.1398:1399](∅→∅),[24.1398]→[24.1398:1399](∅→∅),[24.1399]→[24.3474:3493](∅→∅),[24.3493]→[24.818:877](∅→∅),[24.877]→[24.2:38](∅→∅),[24.38]→[24.915:955](∅→∅),[24.915]→[24.915:955](∅→∅)
  − Command completion should reveal a number of command-line tools from Hydra:

  −

  − <screen>

  − hydra-build hydra-init hydra-update-gc-roots

  − hydra-eval-jobs hydra-queue-runner

  − hydra-evaluator hydra-server

  − </screen>
  [24.3391]

  [24.3494]
  + Command completion should reveal a number of command-line tools

  + from Hydra, such as <command>hydra-queue-runner</command>.
• file deletion: hydra_build.pl (----------)Build.pl (----------)hydra-build (---r------)hydra_build.pl (---r------)hydra_build.pl (---r------)build.pl (----------)hydra_build.pl.in (---r------)
  [24.1560]→[24.0:38](∅→∅),[24.38]→[24.2912:2912](∅→∅),[24.1]→[24.22:54](∅→∅),[24.54]→[24.2912:2912](∅→∅),[24.2543]→[24.196:231](∅→∅),[24.231]→[24.2912:2912](∅→∅),[24.1560]→[24.1892:1930](∅→∅),[24.1930]→[24.2912:2912](∅→∅),[24.2543]→[24.2562:2600](∅→∅),[24.2600]→[24.2912:2912](∅→∅),[24.4]→[24.7550:7582](∅→∅),[24.7582]→[24.2912:2912](∅→∅),[24.2543]→[24.2198888:2198929](∅→∅),[24.2198929]→[24.2912:2912](∅→∅)
  − #! /var/run/current-system/sw/bin/perl

  −

  − use strict;

  − use List::MoreUtils qw(all);

  − use File::Basename;

  − use File::stat;

  − use Hydra::Schema;

  − use Hydra::Helper::Nix;

  − use Hydra::Helper::PluginHooks;

  − use Hydra::Model::DB;

  − use Hydra::Helper::AddBuilds;

  − use Set::Scalar;

  −

  − STDOUT->autoflush();

  −

  − my $db = Hydra::Model::DB->new();

  −

  − my $config = getHydraConfig();

  −

  − my @plugins = Hydra::Plugin->instantiate(db => $db, config => $config);

  −

  −

  − sub addBuildStepOutputs {

  − my ($step) = @_;

  − my $drv = derivationFromPath($step->drvpath);

  − $step->buildstepoutputs->create({ name => $_, path => $drv->{outputs}->{$_} })

  − foreach keys %{$drv->{outputs}};

  − }

  −

  −

  − sub failDependents {

  − my ($drvPath, $status, $errorMsg, $dependents, $startTime, $stopTime, $machine, $propagatedFrom) = @_;

  −

  − # Get the referrer closure of $drvPath.

  − my $dependentDrvs = Set::Scalar->new(computeFSClosure(1, 0, $drvPath));

  −

  − my $time = time();

  −

  − txn_do($db, sub {

  −

  −

  − for my $d (@dependentBuilds) {

  − print STDERR "failing dependent build ", $d->id, " of ", $d->project->name, ":", $d->jobset->name, ":", $d->job->name, "\n";

  − $d->update(

  − { finished => 1

  − , logfile => ''

  − , iscachedbuild => 0

  − , buildstatus => $drvPath eq $d->drvpath ? 1 : 2

  − , starttime => $time

  − , stoptime => $time

  − , errormsg => undef

  − });

  −

  − my $step = $d->buildsteps->create(

  − { stepnr => nextFreeStepNr($d)

  − , type => 0 # = build

  − , drvpath => $drvPath

  − , busy => 0

  − , status => $status

  − , starttime => $startTime

  − , stoptime => $stopTime

  − , errormsg => $errorMsg

  − });

  − addBuildStepOutputs($step);

  − }

  −

  − });

  − }

  −

  −

  −

  − push @$dependents, $d;

  − , machine => $machine

  − , propagatedfrom => $propagatedFrom->id

  − next unless $dependentDrvs->has($d->drvpath);

  − my @dependentBuilds = $db->resultset('Builds')->search(

  − { finished => 0, busy => 0 },

  − { columns => ["id", "project", "jobset", "job", "drvpath", "finished", "busy"] });

  − sub doBuild {

  − my ($build) = @_;

  −

  − my %outputs;

  − $outputs{$_->name} = $_->path foreach $build->buildoutputs->all;

  −

  − my $drvPath = $build->drvpath;

  − my $maxsilent = $build->maxsilent;

  − my $timeout = $build->timeout;

  −

  − my $isCachedBuild = 1;

  − my $outputCreated = 1; # i.e., the Nix build succeeded (but it could be a positive failure)

  − my $startTime = time();

  − my $stopTime = undef;

  −

  − my $buildStatus = 0; # = succeeded

  −

  − my $errormsg = undef;

  −

  − unless (all { isValidPath($_) } values(%outputs)) {

  − $isCachedBuild = 0;

  −

  − # Do the build.

  − my $thisBuildFailed = 0;

  − my $someBuildFailed = 0;

  −

  − # Run Nix to perform the build, and monitor the stderr output

  − # to get notifications about specific build steps, the

  − # associated log files, etc.

  − my $cmd = "nix-store --realise $drvPath " .

  − "--timeout $timeout " .

  − "--max-silent-time $maxsilent " .

  − "--option build-max-log-size 67108864 " .

  − "--keep-going --fallback " .

  − "--no-build-output --log-type flat --print-build-trace " .

  − "--add-root " . gcRootFor($outputs{out} // $outputs{(sort keys %outputs)[0]}) . " 2>&1";

  −

  − my $buildStepNr = nextFreeStepNr($build);

  − my %buildSteps;

  −

  − open OUT, "$cmd |" or die;

  −

  − while (<OUT>) {

  − unless (/^@\s+/) {

  − print STDERR "$_";

  − next;

  − }

  −

  − if (/^@\s+build-started\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)$/) {

  − my $drvPathStep = $1;

  − txn_do($db, sub {

  − my $step = $build->buildsteps->create(

  − { stepnr => ($buildSteps{$drvPathStep} = $buildStepNr++)

  − , type => 0 # = build

  − , drvpath => $drvPathStep

  − , system => $3

  − , busy => 1

  − });

  − addBuildStepOutputs($step);

  − });

  − }

  −

  − elsif (/^@\s+build-remote\s+(\S+)\s+(\S+)$/) {

  − my $drvPathStep = $1;

  − my $machine = $2;

  − txn_do($db, sub {

  − my $step = $build->buildsteps->find({stepnr => $buildSteps{$drvPathStep}}) or die;

  − $step->update({machine => $machine});

  − });

  − }

  −

  − elsif (/^@\s+build-remote-start\s+(\S+)\s+/) {

  − my $drvPathStep = $1;

  − txn_do($db, sub {

  − my $step = $build->buildsteps->find({stepnr => $buildSteps{$drvPathStep}}) or die;

  − $step->update({starttime => time});

  − });

  − }

  −

  − elsif (/^@\s+build-remote-done\s+(\S+)\s+/) {

  − my $drvPathStep = $1;

  − txn_do($db, sub {

  − my $step = $build->buildsteps->find({stepnr => $buildSteps{$drvPathStep}}) or die;

  − $step->update({stoptime => time});

  − });

  − }

  −

  − elsif (/^@\s+build-succeeded\s+(\S+)\s+(\S+)$/) {

  − my $drvPathStep = $1;

  − txn_do($db, sub {

  − my $step = $build->buildsteps->find({stepnr => $buildSteps{$drvPathStep}}) or die;

  − $step->update({busy => 0, status => 0, stoptime => time});

  − $step->update({stoptime => time}) unless defined $step->update;

  − });

  − }

  −

  − elsif (/^@\s+build-failed\s+(\S+)\s+(\S+)\s+(\S+)\s+(.*)$/) {

  − my $drvPathStep = $1;

  − $someBuildFailed = 1;

  − $thisBuildFailed = 1 if $drvPath eq $drvPathStep;

  − my $errorMsg;

  − my $status = 1;

  − if ($3 eq "cached") {

  − $status = 8;

  − } elsif ($3 eq "timeout") {

  − $status = 7;

  − } else {

  − $errorMsg = $4;

  − }

  − my $now = time;

  − my $stepStartTime = $now;

  − my $stepStopTime = $now;

  − my $machine = "";

  − txn_do($db, sub {

  − if ($buildSteps{$drvPathStep}) {

  − my $step = $build->buildsteps->find({stepnr => $buildSteps{$drvPathStep}}) or die;

  − $stepStartTime = $step->starttime;

  − $stepStopTime = $now;

  − $machine = $step->machine;

  − $step->update({busy => 0, status => $status, errormsg => $errorMsg, stoptime => $now});

  − }

  − # Don't write a record if this derivation already

  − # failed previously. This can happen if this is a

  − # restarted build.

  − elsif (scalar $build->buildsteps->search({drvpath => $drvPathStep, type => 0, busy => 0, status => 1}) == 0) {

  − my $step = $build->buildsteps->create(

  − { stepnr => ($buildSteps{$drvPathStep} = $buildStepNr++)

  − , type => 0 # = build

  − , drvpath => $drvPathStep

  − , busy => 0

  − , status => $status

  − , starttime => $now

  − , stoptime => $now

  − , errormsg => $errorMsg

  − });

  − addBuildStepOutputs($step);

  − }

  − });

  −

  − # Immediately fail all builds that depend on this derivation.

  − failDependents($drvPathStep, $status, $errorMsg, $dependents, $stepStartTime, $stepStopTime, $machine, $build);

  − }

  −

  − elsif (/^@\s+substituter-started\s+(\S+)\s+(\S+)$/) {

  − my $path = $1;

  − txn_do($db, sub {

  − my $step = $build->buildsteps->create(

  − { stepnr => ($buildSteps{$path} = $buildStepNr++)

  − , type => 1 # = substitution

  − , busy => 1

  − , starttime => time

  − });

  − # "out" is kinda fake (substitutions don't have named outputs).

  − $step->buildstepoutputs->create({ name => "out", path => $path });

  − });

  − }

  −

  − elsif (/^@\s+substituter-succeeded\s+(\S+)$/) {

  − my $path = $1;

  − txn_do($db, sub {

  − my $step = $build->buildsteps->find({stepnr => $buildSteps{$path}}) or die;

  − $step->update({busy => 0, status => 0, stoptime => time});

  − });

  − }

  − }

  −

  − close OUT;

  −

  − my $res = $?;

  −

  − $stopTime = time();

  −

  − if ($res != 0) {

  − if ($thisBuildFailed) { $buildStatus = 1; }

  − elsif ($someBuildFailed) { $buildStatus = 2; }

  − else { $buildStatus = 3; }

  − }

  −

  − # Only store the output of running Nix if we have a miscellaneous error.

  − $errormsg = undef unless $buildStatus == 3;

  − }

  −

  − done:

  −

  − txn_do($db, sub {

  − if ($buildStatus == 0) {

  −

  − my $size = 0;

  − my $closureSize = 0;

  − my $releaseName;

  −

  − my @closure = computeFSClosure(0, 0, values %outputs);

  − foreach my $path (@closure) {

  − my ($deriver, $hash, $time, $narSize, $refs) = queryPathInfo($path, 0);

  − $closureSize += $narSize;

  − $size += $narSize if grep { $path eq $_ } values(%outputs);

  − }

  −

  − foreach my $path (values %outputs) {

  − $buildStatus = 6 if $buildStatus == 0 && -f "$path/nix-support/failed";

  − $releaseName //= getReleaseName($path);

  − }

  −

  − $build->update(

  − { releasename => $releaseName

  − , size => $size

  − , closuresize => $closureSize

  − });

  −

  − addBuildProducts($db, $build);

  − }

  −

  − $build->update(

  − { finished => 1

  − , busy => 0

  − , locker => ''

  − , logfile => ''

  − , iscachedbuild => $isCachedBuild

  − , buildstatus => $buildStatus

  − , starttime => $startTime

  − , stoptime => $stopTime // time()

  − , errormsg => $errormsg

  − });

  −

  − });

  − }

  −

  −

  − my $buildId = $ARGV[0] or die "syntax: $0 BUILD-ID\n";

  − print STDERR "performing build $buildId\n";

  −

  − # Lock the build. If necessary, steal the lock from the parent

  − # process (runner.pl). This is so that if the runner dies, the

  − # children (i.e. the build.pl instances) can continue to run and won't

  − # have the lock taken away.

  − my $build;

  − txn_do($db, sub {

  − $build = $db->resultset('Builds')->find($buildId);

  − die "build $buildId doesn't exist\n" unless defined $build;

  − die "build $buildId already done\n" if $build->finished;

  − if ($build->busy != 0 && $build->locker != getppid) {

  − die "build $buildId is already being built";

  − }

  − $build->update({busy => 1, locker => $$});

  − $build->buildsteps->search({busy => 1})->delete;

  − $build->buildproducts->delete;

  − });

  −

  − die unless $build;

  −

  − # Do the build. If it throws an error, unlock the build so that it

  − # can be retried.

  − eval {

  − doBuild $build;

  − print "done\n";

  − };

  − if ($@) {

  − warn $@;

  − txn_do($db, sub {

  − $build->update({busy => 0, locker => $$});

  − });

  − }

  − if ($ENV{'HYDRA_MAIL_TEST'}) {

  − my $build = $db->resultset('Builds')->find($buildId);

  − notifyBuildFinished(\@plugins, $build, []);

  − exit 0;

  − }

  −

  −

  − notifyBuildFinished(\@plugins, $build, $dependents);

  − # Mark any remaining active build steps as aborted.

  − $build->buildsteps->search({ busy => 1 })->update({ busy => 0, status => 4, stoptime => time });

  −

  −

  − elsif (/^@\s+substituter-failed\s+(\S+)\s+(\S+)\s+(\S+)$/) {

  − my $path = $1;

  − txn_do($db, sub {

  − my $step = $build->buildsteps->find({stepnr => $buildSteps{$path}}) or die;

  − $step->update({busy => 0, status => 1, errormsg => $3, stoptime => time});

  − });

  − }

  −

  − else {

  − print STDERR "unknown Nix trace message: $_";

  − }

  − , starttime => time

  − $errormsg .= $_;

  −

  − "--option print-missing false " .

  − if (!isValidPath($drvPath)) {

  − $buildStatus = 3;

  − $errormsg = "derivation was garbage-collected prior to build";

  − goto done;

  − }

  −

  −

  − my $dependents = [];

  − }

  −

  −

  − sub nextFreeStepNr {

  − my ($build) = @_;

  − my $max = $build->buildsteps->find(

  − {}, {select => {max => 'stepnr + 1'}, as => ['max']});

  − return (defined $max && defined $max->get_column('max')) ? $max->get_column('max') : 1;

  − use Nix::Store;

  − use Hydra::Plugin;
• file deletion: hydra_queue_runner.pl (----------)Runner.pl (----------)hydra-queue-runner (---r------)hydra_queue_runner.pl (---r------)hydra_queue_runner.pl (---r------)runner.pl (----------)hydra_queue_runner.pl.in (---r------)
  [24.1560]→[24.38:83](∅→∅),[24.83]→[24.7823:7823](∅→∅),[24.1]→[24.154:187](∅→∅),[24.187]→[24.7823:7823](∅→∅),[24.2543]→[24.485:527](∅→∅),[24.527]→[24.7823:7823](∅→∅),[24.1560]→[24.1973:2018](∅→∅),[24.2018]→[24.7823:7823](∅→∅),[24.2543]→[24.2792:2837](∅→∅),[24.2837]→[24.7823:7823](∅→∅),[24.4]→[24.9347:9380](∅→∅),[24.9380]→[24.7823:7823](∅→∅),[24.2543]→[24.2199118:2199166](∅→∅),[24.2199166]→[24.7823:7823](∅→∅)
  − #! /var/run/current-system/sw/bin/perl

  −

  − use strict;

  − use Cwd;

  − use File::Basename;

  − use POSIX qw(dup2 :sys_wait_h);

  − use Hydra::Schema;

  − use Hydra::Helper::Nix;

  − use Hydra::Model::DB;

  − use IO::Handle;

  − use Nix::Store;

  − use Set::Scalar;

  −

  − chdir Hydra::Model::DB::getHydraPath or die;

  − my $db = Hydra::Model::DB->new();

  −

  − STDOUT->autoflush();

  −

  − my $lastTime;

  −

  − #$SIG{CHLD} = 'IGNORE';

  −

  −

  − sub unlockDeadBuilds {

  − # Unlock builds whose building process has died.

  − txn_do($db, sub {

  − my @builds = $db->resultset('Builds')->search({finished => 0, busy => 1});

  − foreach my $build (@builds) {

  − my $pid = $build->locker;

  − my $unlock = 0;

  − if ($pid == $$) {

  − if (!defined $lastTime || $build->starttime < $lastTime - 300) {

  − $unlock = 1;

  − }

  − } elsif (kill(0, $pid) != 1) { # see if we can signal the process

  − $unlock = 1;

  − }

  − if ($unlock) {

  − print "build ", $build->id, " pid $pid died, unlocking\n";

  − $build->update({ busy => 0, locker => "" });

  − $build->buildsteps->search({ busy => 1 })->update({ busy => 0, status => 4, stoptime => time });

  − }

  − }

  − });

  − }

  −

  −

  − sub findBuildDependencyInQueue {

  − my ($buildsByDrv, $build) = @_;

  − return undef unless isValidPath($build->drvpath);

  − my @deps = grep { /\.drv$/ && $_ ne $build->drvpath } computeFSClosure(0, 0, $build->drvpath);

  − return unless scalar @deps > 0;

  − foreach my $d (@deps) {

  − my $bs = $buildsByDrv->{$d};

  − next unless defined $bs;

  − return $db->resultset('Builds')->find((@$bs)[0]);

  − }

  − return undef;

  − }

  −

  − sub checkBuilds {

  − # print "looking for runnable builds...\n";

  −

  − my @buildsStarted;

  −

  − my $machines = getMachines;

  −

  − txn_do($db, sub {

  −

  − # Cache scheduled builds by derivation path to speed up

  − # findBuildDependencyInQueue.

  − my $buildsByDrv = {};

  − push @{$buildsByDrv->{$_->drvpath}}, $_->id

  − foreach $db->resultset('Builds')->search({ finished => 0 });

  −

  − # Builds in the queue of which a dependency is already building.

  − my $blockedBuilds = Set::Scalar->new();

  − blockBuilds($buildsByDrv, $blockedBuilds, $_)

  − foreach $db->resultset('Builds')->search({ finished => 0, busy => 1 });

  −

  − # Get the system types for the runnable builds.

  − my @systemTypes = $db->resultset('Builds')->search(

  − { finished => 0, busy => 0 },

  − { join => ['project'], select => ['system'], as => ['system'], distinct => 1 });

  −

  − # Get the total number of scheduling shares.

  − my $totalShares = getTotalShares($db) || 1;

  −

  − # For each system type, select up to the maximum number of

  − # concurrent build for that system type.

  − foreach my $system (@systemTypes) {

  − # How many builds are already currently executing for this

  − # system type?

  − my $nrActive = $db->resultset('Builds')->search(

  − {finished => 0, busy => 1, system => $system->system})->count;

  −

  − (my $systemTypeInfo) = $db->resultset('SystemTypes')->search({system => $system->system});

  − my $max = defined $systemTypeInfo ? $systemTypeInfo->maxconcurrent : $maxConcurrent{$system->system} // 2;

  −

  − my $extraAllowed = $max - $nrActive;

  − next if $extraAllowed <= 0;

  −

  − print STDERR "starting at most $extraAllowed builds for system ${\$system->system}\n";

  −

  − j: while ($extraAllowed-- > 0) {

  −

  − my @runnableJobsets = $db->resultset('Builds')->search(

  − { finished => 0, busy => 0, system => $system->system },

  − { select => ['project', 'jobset'], distinct => 1 });

  −

  − next if @runnableJobsets == 0;

  −

  − my $windowSize = 24 * 3600;

  − my $totalWindowSize = $windowSize * $max;

  −

  − my @res;

  −

  − foreach my $b (@runnableJobsets) {

  − my $jobset = $db->resultset('Jobsets')->find($b->get_column('project'), $b->get_column('jobset')) or die;

  −

  − my $timeSpent = $timeSpentPerJobset->{$b->get_column('project')}->{$b->get_column('jobset')};

  −

  − if (!defined $timeSpent) {

  − $timeSpent = $jobset->builds->search(

  − { },

  − { where => \ ("(finished = 0)")

  − , join => 'buildsteps'

  − , select => \ "sum(coalesce(buildsteps.stoptime, ${\time}) - buildsteps.starttime)"

  − , as => "sum" })->single->get_column("sum") // 0;

  −

  − $timeSpent += $jobset->builds->search(

  − { },

  − { where => \ ("(me.stoptime >= " . (time() - $windowSize) . ")")

  − , join => 'buildsteps'

  − , select => \ "sum(coalesce(buildsteps.stoptime, ${\time}) - buildsteps.starttime)"

  − , as => "sum" })->single->get_column("sum") // 0;

  −

  − # Add a 30s penalty for each started build. This

  − # is to account for jobsets that have running

  − # builds but no build steps yet.

  − $timeSpent += $jobset->builds->search({ finished => 0, busy => 1 })->count * $costPerBuild;

  −

  − $timeSpentPerJobset->{$b->get_column('project')}->{$b->get_column('jobset')} = $timeSpent;

  − }

  −

  − my $share = $jobset->schedulingshares || 1; # prevent division by zero

  − my $used = $timeSpent / ($totalWindowSize * ($share / $totalShares));

  −

  − #printf STDERR "%s:%s: %d s, total used = %.2f%%, share used = %.2f%%\n", $jobset->get_column('project'), $jobset->name, $timeSpent, $timeSpent / $totalWindowSize * 100, $used * 100;

  −

  − push @res, { jobset => $jobset, used => $used };

  − }

  −

  − foreach my $r (sort { $a->{used} <=> $b->{used} } @res) {

  − my $jobset = $r->{jobset};

  − #print STDERR "selected ", $jobset->get_column('project'), ':', $jobset->name, "\n";

  −

  − # Select the highest-priority build for this jobset.

  − my @builds = $jobset->builds->search(

  − { finished => 0, busy => 0, system => $system->system },

  − { order_by => ["priority DESC", "id"] });

  −

  − foreach my $build (@builds) {

  − # Find a dependency of $build that has no queued

  − # dependencies itself. This isn't strictly necessary,

  − # but it ensures that Nix builds are done as part of

  − # their corresponding Hydra builds, rather than as a

  − # dependency of some other Hydra build.

  − while (my $dep = findBuildDependencyInQueue($buildsByDrv, $build)) {

  − $build = $dep;

  − }

  − next if $build->busy;

  −

  − printf STDERR "starting build %d (%s:%s:%s) on %s; jobset at %.2f%% of its share\n",

  − $build->id, $build->project->name, $build->jobset->name, $build->job->name, $build->system, $r->{used} * 100;

  −

  − my $logfile = getcwd . "/logs/" . $build->id;

  − mkdir(dirname $logfile);

  − unlink($logfile);

  − $build->update(

  − { busy => 1

  − , locker => $$

  − , logfile => $logfile

  − });

  − push @buildsStarted, $build;

  − next j;

  − }

  − }

  −

  − last; # nothing found, give up on this system type

  − }

  − }

  −

  − $lastTime = time();

  −

  − $_->update({ starttime => time() }) foreach @buildsStarted;

  − });

  −

  − # Actually start the builds we just selected. We need to do this

  − # outside the transaction in case it aborts or something.

  − foreach my $build (@buildsStarted) {

  − my $id = $build->id;

  − eval {

  − my $logfile = $build->logfile;

  − my $child = fork();

  − die unless defined $child;

  − if ($child == 0) {

  − eval {

  − open LOG, ">$logfile" or die "cannot create logfile $logfile";

  − POSIX::dup2(fileno(LOG), 1) or die;

  − POSIX::dup2(fileno(LOG), 2) or die;

  − exec("hydra-build", $id);

  − };

  − warn "cannot start build $id: $@";

  − POSIX::_exit(1);

  − }

  − };

  − if ($@) {

  − warn $@;

  − txn_do($db, sub {

  − $build->update({ busy => 0, locker => $$ });

  − });

  − }

  − }

  −

  − while (1) {

  − eval {

  − unlockDeadBuilds;

  −

  − checkBuilds;

  − };

  − warn $@ if $@;

  −

  − # print "sleeping...\n";

  − sleep(5);

  − }

  − # Clean up zombies.

  − while ((waitpid(-1, &WNOHANG)) > 0) { };

  −

  − }

  −

  −

  − if (scalar(@ARGV) == 1 && $ARGV[0] eq "--unlock") {

  − unlockDeadBuilds;

  − exit 0;

  − }

  −

  −

  − $timeSpentPerJobset->{$jobset->get_column('project')}->{$jobset->name} += $costPerBuild;

  −

  − blockBuilds($buildsByDrv, $blockedBuilds, $build);

  −

  − next if $blockedBuilds->has($build->id);

  −

  − my $costPerBuild = 30;

  −

  − my $timeSpentPerJobset;

  − my %maxConcurrent;

  −

  − foreach my $machineName (keys %{$machines}) {

  − foreach my $system (@{${$machines}{$machineName}{'systemTypes'}}) {

  − $maxConcurrent{$system} = (${$machines}{$machineName}{'maxJobs'} or 0) + ($maxConcurrent{$system} or 0)

  − }

  − }

  −

  −

  − sub blockBuilds {

  − my ($buildsByDrv, $blockedBuilds, $build) = @_;

  − my @rdeps = grep { /\.drv$/ && $_ ne $build->drvpath } computeFSClosure(1, 0, $build->drvpath);

  − foreach my $drv (@rdeps) {

  − my $bs = $buildsByDrv->{$drv};

  − next if !defined $bs;

  − $blockedBuilds->insert($_) foreach @$bs;

  − }

  − }

  −

  −

  − # Given a build, return an arbitrary queued build on which this build

  − # depends; or undef if no such build exists.
• edit in src/script/Makefile.am at line 7
  [24.422]→[24.422:442](∅→∅)
  − hydra-build \
• edit in src/script/Makefile.am at line 8
  [24.465]→[24.465:491](∅→∅)
  − hydra-queue-runner \
• replacement in src/script/hydra-eval-guile-jobs.in at line 107
  [17.438]→[17.438:504](∅→∅)
  − ;; the time 'hydra-build' attempts to build it.
  [17.438]

  [17.504]
  + ;; the time 'hydra-queue-runner' attempts to build it.