Remove hydra-build and the old hydra-queue-runner

[?]

Jun 22, 2015, 1:43 PM

UXRNODRJJU7A33F4PG24WOHKTJFI3XQMZGT2MUOBRAH2MPQ7NZ2AC

Dependencies

[2] MREXL5RT Whoops
[3] Z4Y3TVEE hydra-queue-{runner,evaluator}: don't clutter the system log with debug messages
[4] ZNGKTEAI If a build aborts, mark any remaining active build steps as aborted
[5] 6SGBF7JP Set build status to 1 if the primary build failed
[6] WMSCRPSC Fill in starttime/stoptime for cached builds
[7] 6PG5BATL Remove tabs
[8] PHNLYPKB Call buildFinished when a cached build is added
[9] VXTPW6BF hydra-build: Hack to handle timeouts
[10] FUJ3A66L hydra-queue-runner: Set the start time properly
[11] MMUBEIGV hydra-queue-runner: Tweaked the selection method
[12] IEP2Q6GV *headdesk*
[13] TSR3RZC4 Revert "Add a dependency_lookup configuration option to enable (slow) dependency lookup in queue. This behaviour was disabled temporarily in accefbb79 due to slowness in very large queues, but some people might be dependent on it, so it is configurable until the previous behaviour is implemented more efficiently."
[14] AWMM5OGV Use delete instead of delete_all
[15] ZZYTUBG2 Fix extreme slowness in hydra-queue-runner
[16] 533PIYC3 Prevent a division by zero in hydra-queue-runner
[17] IS7GUIWY hydra-eval-guile-jobs: Register derivations as GC roots.
[18] COLZ7V4E hydra-build: Handle new trace messages
[19] 3YHNO5H2 Don't use Perl's -w flag
[20] UJOSXBZI Record which build a failed build step was propagated from
[21] E746D2DR Don't show missing paths in logs
[22] WJ7IKFXL Fix not-null constraint violation inserting build step
[23] QMC6IMJQ Split timeSpent query into 2 separate queries, as postgresql isn't able to figure out a decent query plan. With 120k jobs in queue, this makes some queries go from 100s to 1-2s.
[24] RNNLIVYY Respect SystemTypes if defined
[25] MNZ67UXO If a build step fail, immediately fail all queued builds that depend on it
[26] A32UT34U hydra_build.pl: Honor `$build->timeout'.
[27] YTESD75Z * Get rid of zombies.
[28] A63IHCMX * Register GC roots properly.
[29] 3BKF6P72 * Use Nix's negative caching.
[30] RWIBJ5L4 * Autoflush stdout.
[31] MOX7XJ2E Merge the BuildSchedulingInfo table into the Builds table
[32] R6B5CAFF Let Builds.timestamp refer to the time the build was added
[33] RBHHV7P7 * Read logs using logContents function in stead of handling it everywhere separately.
[34] 3PNG7NIB Remove trailing whitespace
[35] MPGVCHVF * Fix an apparent incompatibility with recent DBIx::Class.
[36] TFLAR4KA hydra-queue-runner: Start as many builds as possible on each iteration
[37] 2KDHXY52 Kill builds that produce more than 64 MiB of log output
[38] QMW24O5S Add support for Guile & Guix.
[39] 57Y7RG2A Update queryPathInfo calls
[40] KBW3FDZ2 Merge remote branch 'remotes/origin/master'
[41] AHTEIK7G * Added a maintainers field to the Builds table.
[42] 37R34XJO * Negative caching: don't perform a build if a dependency already
[43] EFWN7JBV * Added a status page that shows all the currently executing build steps.
[44] IJPTEKRF
[45] L4AI5YL6 Rename hydra_*.pl to hydra-*
[46] HE3GX5IP Optimize fetch-git.
[47] VI32YSGA I should test first
[48] GEADFVZ5 hydra-queue-runner: Improved scheduling
[49] CQJX3RGU hydra-queue-runner: Don't kill builds we just started
[50] UCVYVMXB Use the same start/stop time for the build steps as for the build
[51] JUTAEI7R Restore old findBuildDependencyInQueue behaviour
[52] UGA45FNC Add a plugin for backing up builds in s3
[53] XHOZT4WT Add a command `hydra-create-user' for managing user accounts
[54] GAIBDEZZ * Store the name of the machine that performed a build step in the
[55] L2E6EVE2 * Merged the Build and Job tables.
[56] WOHG5MJD hydra: catalyst plain view doesn;t work with data => 0, so put some whitespace around it
[57] 5DSF5KWY * Perl sucks. "getBuildLog $drvPath" doesn't mean the same as
[58] JR76VU3W Remove timeout detection hack
[59] QUTWJR7P * Include more info in notification emails.
[60] EUWLW7FY Don't use the Switch module
[61] UN2KZL3A Rename c -> hydra-eval-jobs
[62] A22P7HCO hydra: at evaluation, check if path is already built, and mark as built in stead of adding to the queue.
[63] Q6SOGMDQ Hydra/28: Rename "scheduler" to "evaluator"
[64] 4SJPAAJX Pass failing dependent builds to buildFinished
[65] CLJQCY2X * Store info about all the build actions and allow them to be
[66] 3E6IP3R3 * Add the name of the jobset to ReleaseSetJobs, otherwise we can't
[67] GYPHTT4M Manual: Remove tabs, indent consistently
[68] 6US6LEC7 * Add a NarSize field to Hydra manifests. This allows nix-env
[69] FHF6IZJQ * Basic release management: releases are now dynamically computed as
[70] ULHEOJO2 * Better timeout.
[71] B72GLND4
[72] PKPWUHUX * Idem.
[73] NLJJZVHO * Use ->update({...}) properly.
[74] JGLE5BRN Add separate build step status codes for cached failures and timeouts
[75] 4LR7CSBT hydra-build: Give a nicer error message if the derivation is gone
[76] NREF6YOA * Don't start more builds concurrently than allowed for each system
[77] T4LLYESZ * Nix expression for building Hydra.
[78] EKNK5AHQ doc: Augment the "Installation" section with material by Visser & Dolstra.
[79] D7TT2BNK
[80] UMOJJ6DV time out to 3600 for now
[81] BOFOHCPK removed debug print, added last 50 lines in failure emails
[82] KX5L74EY add nix-prefetch- scripts for now, were externals in svn
[83] TJK27WSB Open the DB using Hydra::Model::DB->new
[84] WQXF2T3D hydra-evaluator: Don't require $HYDRA_CONFIG
[85] FHAVPTZ6 Hydra/23: added some X-headers with meta info in email notifications, added more descriptive status
[86] 2PWOXJTX
[87] SIZ6KXPW Set the build status properly for failing local builds
[88] 7VWDMKAZ hydra-queue-runner: don't clutter the system log with debug message
[89] S3ZLZP3N * Cut off builds after half an hour of apparent inactivity. This
[90] QAJK5MCE Remove obsolete hydra-control script
[91] X27GNHDV * Basic job info in the database.
[92] WHAFVCEI
[93] SX5XYD6I When propagating failure, propagate the duration and machine
[94] WM5WAST3 * Monitor for dead builds all the time.
[95] 64K7R4Y6 Forgot to change Nix to Nix::Store in one place
[96] LQNBKF3D
[97] VJP6O6WA * Doh. Remove debug statement.
[98] DN3VAAP4 hydra: remove HYDRA_HOME env var, not used
[99] TPSCSZKX Speed up findBuildDependencyInQueue
[100] ERNOO5ZZ * Reorganising.
[101] 24BMQDZA Start of single-process hydra-queue-runner
[102] DGZE7ZRA * Don't log redundant build steps in case of cached failures.
[103] FYO6NECE hydra
[104] J5UVLXOK * Start of a basic Catalyst web interface.
[105] G5A7TZVI * Don't discard old build steps when restarting a build.
[106] AZ4LR2GT * Scripts for starting / stopping Hydra.
[107] ECBA3GQO * Make the schema class names match the case of the SQL table names.
[108] D5QIOJGP * Move everything up one directory.
[109] Y6AHH4TH Remove the logfile and logSize columns from the database
[110] 6MGFQDR2 deleted some old scripts
[111] DQD7JMSU * Fix the terminology.
[112] YEXD7CBK Fix findBuildDependencyInQueue
[113] CXRCPDSQ * added support for twitter notification
[114] A5V7HLIQ * Shut up DBIx::Class.
[115] ZTQEU5QS Hydra: Add support for maxSilent meta attribute (also already added timeout, but not implemented the actual timeout for the build yet)
[116] HPEG2RHV Merge the BuildResultInfo table into the Builds table
[117] FV2M6MOT hydra: use autoconf/-make
[118] IUCHXUJP Use "can" to check whether a plugin supports buildFinished
[119] N22GPKYT * Put info about logs / build products in the DB.
[120] WZ3AEJ67 * hydra_update_gc_roots.pl registers build outputs that should be kept
[121] BLVQGJ4L Use OO-style plugins
[122] 2T42QGZD * Register builds as GC roots so they don't get deleted.
[123] EX4FXA5T Handle active build steps of aborted builds properly
[124] 5EQYVRWE Add a plugin mechanism
[125] ARD6Z67T Do incremental SVN checkouts
[126] DTXTS7LN * Speed up findBuildDependencyInQueue by doing only one SQL query for
[127] 2IEFMER5 * Add --fallback to prevent problems with obsolete substitutes.
[128] KA45EBF5 * Send email if a build fails.
[129] AS5PAYLI
[130] US27ZTX5 * HydraFrontend -> Hydra.
[131] SA5ZZ3I4 hydra-queue-runner: Use nix.machines instead of the SystemTypes table to determine how many build jobs are allowed per system type.
[132] I7UELKBV hydra-queue-runner: Don't unlock builds we just started
[133] TULPZ62Y * Perform builds in parallel.
[134] CURCK6C2
[135] GZAXDYBQ * Script for initialising the database.
[136] 7YBYT2LQ
[137] TWVSALRL * Allow the maximum number of concurrent builds per platform to be
[138] LZO3C2KI * Hack around those SQLite timeouts: just retry the transaction.
[139] 7DWCXNC7 Use the new Nix Perl bindings
[140] XYBSTHMJ fix for buildsteps starting at 0, probably something changed in catalyst
[141] QI7MGZAU
[142] Y6H7Y3OT Capture the path to `guile', when available.
[143] K3EAQY3X * Doh.
[144] M2CFFNJY Remove unused file
[145] D7NXMCON * Doh.
[146] OG7BEM57
[147] R5D7DZPE
[148] D6YQQQCN * Don't ignore SIGCHLD after all, Perl doesn't like it. Just do
[149] W5GW4E22 Don't install hydra-create
[150] OV7F5M3E Merge branch 'queue-17'
[151] CLXEECMF * Start putting build results in a database.
[152] T7CCJQOF * Revert for now due to Postgres breakage.
[153] AMFMXR52 Provide a command ‘hydra-init’ to initialise/upgrade the database
[154] 66MEE6QG hydra-build: Don't send a giant query to the database
[155] JK2QWPH6
[156] NB2VOKIR Include names of committers in HipChat notifications
[157] LSPRR4TK hydra-queue-runner: Disable findBuildDependencyInQueue for now
[158] 4N5APGRG * Start of a helper tool to evaluate job expressions efficiently.
[159] J4AJQ6RX Add a dependency_lookup configuration option to enable (slow) dependency lookup in queue. This behaviour was disabled temporarily in accefbb79 due to slowness in very large queues, but some people might be dependent on it, so it is configurable until the previous behaviour is implemented more efficiently.
[160] 6KCP6ODP * Get the URI for use in notification mails from the Hydra config
[161] IW2LHCLL fixed email bug
[162] 5WCR3MKU trunk -> master
[163] YTIDBFGU Drop unused "disabled" columns
[164] 6L3ZM55S Add font for the captcha
[165] PMNWRTGJ Add multiple output support
[166] PYTQMQKD Hydra/17: in queue runner, prefer builds in the queue that are a dependency of another build (with higher priority)
[167] CMU3YKOU * Store the release name.
[168] DBPIYHMA hydra: add nix-prefetch-* to tarball
[169] X3YLTWJA hydra-queue-runner: Cache the lookup of time spent per jobset
[170] AFTXA575 * $HYDRA_DATA environment variable.
[171] WTUBAA5Q hydra-queue-runner: Handle restarted builds whose derivation is gone
[172] QZLMDKMU * Queue runner: don't start scheduled builds builds if they belong to
[173] YFPZ46YK * hydra: added variant of build input type, 'build output (same system)' to allow better continous integration in one jobset for multiple system. it makes sure that the system of the build that is passed as input for a job has the same system as the job.
[174] UVNQPK3T Hydra/56: handle failed builds with result only at build level, not buildsteps
[175] 5O6E5SU5 hydra: store logfile/output path/closure size
[176] D3DIBMOK * For products that are directories (like manuals), allow a default
[177] G3IUM7VL hydra: add support for succeedOnFailure feature in stdenv
[178] BD3GRK4B * Get rid of "positive failures" and separate log phases.
[179] FLPZ3YCK Remove debug line
[180] TRFRGOBD Remove Twitter notification support

Change contents

replacement in doc/manual/installation.xml at line 103

[24.3391]→[24.3391:3473](∅→∅),[24.3035]→[24.1398:1399](∅→∅),[24.3473]→[24.1398:1399](∅→∅),[24.1398]→[24.1398:1399](∅→∅),[24.1399]→[24.3474:3493](∅→∅),[24.3493]→[24.818:877](∅→∅),[24.877]→[24.2:38](∅→∅),[24.38]→[24.915:955](∅→∅),[24.915]→[24.915:955](∅→∅)

Command completion should reveal a number of command-line tools from Hydra:

<screen>
hydra-build hydra-init hydra-update-gc-roots
hydra-eval-jobs hydra-queue-runner
hydra-evaluator hydra-server
</screen>

[24.3391]

[24.3494]

Command completion should reveal a number of command-line tools
from Hydra, such as <command>hydra-queue-runner</command>.
file deletion: hydra_build.pl (----------)Build.pl (----------)hydra-build (---r------)hydra_build.pl (---r------)hydra_build.pl (---r------)build.pl (----------)hydra_build.pl.in (---r------)

[24.1560]→[24.0:38](∅→∅),[24.38]→[24.2912:2912](∅→∅),[24.1]→[24.22:54](∅→∅),[24.54]→[24.2912:2912](∅→∅),[24.2543]→[24.196:231](∅→∅),[24.231]→[24.2912:2912](∅→∅),[24.1560]→[24.1892:1930](∅→∅),[24.1930]→[24.2912:2912](∅→∅),[24.2543]→[24.2562:2600](∅→∅),[24.2600]→[24.2912:2912](∅→∅),[24.4]→[24.7550:7582](∅→∅),[24.7582]→[24.2912:2912](∅→∅),[24.2543]→[24.2198888:2198929](∅→∅),[24.2198929]→[24.2912:2912](∅→∅)

#! /var/run/current-system/sw/bin/perl

use strict;
use List::MoreUtils qw(all);
use File::Basename;
use File::stat;
use Hydra::Schema;
use Hydra::Helper::Nix;
use Hydra::Helper::PluginHooks;
use Hydra::Model::DB;
use Hydra::Helper::AddBuilds;
use Set::Scalar;

STDOUT->autoflush();

my $db = Hydra::Model::DB->new();

my $config = getHydraConfig();

my @plugins = Hydra::Plugin->instantiate(db => $db, config => $config);

sub addBuildStepOutputs {
my ($step) = @_;
my $drv = derivationFromPath($step->drvpath);
$step->buildstepoutputs->create({ name => $_, path => $drv->{outputs}->{$_} })
foreach keys %{$drv->{outputs}};
}

sub failDependents {
my ($drvPath, $status, $errorMsg, $dependents, $startTime, $stopTime, $machine, $propagatedFrom) = @_;

# Get the referrer closure of $drvPath.
my $dependentDrvs = Set::Scalar->new(computeFSClosure(1, 0, $drvPath));

my $time = time();

txn_do($db, sub {

for my $d (@dependentBuilds) {
print STDERR "failing dependent build ", $d->id, " of ", $d->project->name, ":", $d->jobset->name, ":", $d->job->name, "\n";
$d->update(
{ finished => 1
, logfile => ''
, iscachedbuild => 0
, buildstatus => $drvPath eq $d->drvpath ? 1 : 2
, starttime => $time
, stoptime => $time
, errormsg => undef
});

my $step = $d->buildsteps->create(
{ stepnr => nextFreeStepNr($d)
, type => 0 # = build
, drvpath => $drvPath
, busy => 0
, status => $status
, starttime => $startTime
, stoptime => $stopTime
, errormsg => $errorMsg
});
addBuildStepOutputs($step);
}

});
}

push @$dependents, $d;
, machine => $machine
, propagatedfrom => $propagatedFrom->id
next unless $dependentDrvs->has($d->drvpath);
my @dependentBuilds = $db->resultset('Builds')->search(
{ finished => 0, busy => 0 },
{ columns => ["id", "project", "jobset", "job", "drvpath", "finished", "busy"] });
sub doBuild {
my ($build) = @_;

my %outputs;
$outputs{$_->name} = $_->path foreach $build->buildoutputs->all;

my $drvPath = $build->drvpath;
my $maxsilent = $build->maxsilent;
my $timeout = $build->timeout;

my $isCachedBuild = 1;
my $outputCreated = 1; # i.e., the Nix build succeeded (but it could be a positive failure)
my $startTime = time();
my $stopTime = undef;

my $buildStatus = 0; # = succeeded

my $errormsg = undef;

unless (all { isValidPath($_) } values(%outputs)) {
$isCachedBuild = 0;

# Do the build.
my $thisBuildFailed = 0;
my $someBuildFailed = 0;

# Run Nix to perform the build, and monitor the stderr output
# to get notifications about specific build steps, the
# associated log files, etc.
my $cmd = "nix-store --realise $drvPath " .
"--timeout $timeout " .
"--max-silent-time $maxsilent " .
"--option build-max-log-size 67108864 " .
"--keep-going --fallback " .
"--no-build-output --log-type flat --print-build-trace " .
"--add-root " . gcRootFor($outputs{out} // $outputs{(sort keys %outputs)[0]}) . " 2>&1";

my $buildStepNr = nextFreeStepNr($build);
my %buildSteps;

open OUT, "$cmd |" or die;

while (<OUT>) {
unless (/^@\s+/) {
print STDERR "$_";
next;
}

if (/^@\s+build-started\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)$/) {
my $drvPathStep = $1;
txn_do($db, sub {
my $step = $build->buildsteps->create(
{ stepnr => ($buildSteps{$drvPathStep} = $buildStepNr++)
, type => 0 # = build
, drvpath => $drvPathStep
, system => $3
, busy => 1
});
addBuildStepOutputs($step);
});
}

elsif (/^@\s+build-remote\s+(\S+)\s+(\S+)$/) {
my $drvPathStep = $1;
my $machine = $2;
txn_do($db, sub {
my $step = $build->buildsteps->find({stepnr => $buildSteps{$drvPathStep}}) or die;
$step->update({machine => $machine});
});
}

elsif (/^@\s+build-remote-start\s+(\S+)\s+/) {
my $drvPathStep = $1;
txn_do($db, sub {
my $step = $build->buildsteps->find({stepnr => $buildSteps{$drvPathStep}}) or die;
$step->update({starttime => time});
});
}

elsif (/^@\s+build-remote-done\s+(\S+)\s+/) {
my $drvPathStep = $1;
txn_do($db, sub {
my $step = $build->buildsteps->find({stepnr => $buildSteps{$drvPathStep}}) or die;
$step->update({stoptime => time});
});
}

elsif (/^@\s+build-succeeded\s+(\S+)\s+(\S+)$/) {
my $drvPathStep = $1;
txn_do($db, sub {
my $step = $build->buildsteps->find({stepnr => $buildSteps{$drvPathStep}}) or die;
$step->update({busy => 0, status => 0, stoptime => time});
$step->update({stoptime => time}) unless defined $step->update;
});
}

elsif (/^@\s+build-failed\s+(\S+)\s+(\S+)\s+(\S+)\s+(.*)$/) {
my $drvPathStep = $1;
$someBuildFailed = 1;
$thisBuildFailed = 1 if $drvPath eq $drvPathStep;
my $errorMsg;
my $status = 1;
if ($3 eq "cached") {
$status = 8;
} elsif ($3 eq "timeout") {
$status = 7;
} else {
$errorMsg = $4;
}
my $now = time;
my $stepStartTime = $now;
my $stepStopTime = $now;
my $machine = "";
txn_do($db, sub {
if ($buildSteps{$drvPathStep}) {
my $step = $build->buildsteps->find({stepnr => $buildSteps{$drvPathStep}}) or die;
$stepStartTime = $step->starttime;
$stepStopTime = $now;
$machine = $step->machine;
$step->update({busy => 0, status => $status, errormsg => $errorMsg, stoptime => $now});
}
# Don't write a record if this derivation already
# failed previously. This can happen if this is a
# restarted build.
elsif (scalar $build->buildsteps->search({drvpath => $drvPathStep, type => 0, busy => 0, status => 1}) == 0) {
my $step = $build->buildsteps->create(
{ stepnr => ($buildSteps{$drvPathStep} = $buildStepNr++)
, type => 0 # = build
, drvpath => $drvPathStep
, busy => 0
, status => $status
, starttime => $now
, stoptime => $now
, errormsg => $errorMsg
});
addBuildStepOutputs($step);
}
});

# Immediately fail all builds that depend on this derivation.
failDependents($drvPathStep, $status, $errorMsg, $dependents, $stepStartTime, $stepStopTime, $machine, $build);
}

elsif (/^@\s+substituter-started\s+(\S+)\s+(\S+)$/) {
my $path = $1;
txn_do($db, sub {
my $step = $build->buildsteps->create(
{ stepnr => ($buildSteps{$path} = $buildStepNr++)
, type => 1 # = substitution
, busy => 1
, starttime => time
});
# "out" is kinda fake (substitutions don't have named outputs).
$step->buildstepoutputs->create({ name => "out", path => $path });
});
}

elsif (/^@\s+substituter-succeeded\s+(\S+)$/) {
my $path = $1;
txn_do($db, sub {
my $step = $build->buildsteps->find({stepnr => $buildSteps{$path}}) or die;
$step->update({busy => 0, status => 0, stoptime => time});
});
}
}

close OUT;

my $res = $?;

$stopTime = time();

if ($res != 0) {
if ($thisBuildFailed) { $buildStatus = 1; }
elsif ($someBuildFailed) { $buildStatus = 2; }
else { $buildStatus = 3; }
}

# Only store the output of running Nix if we have a miscellaneous error.
$errormsg = undef unless $buildStatus == 3;
}

done:

txn_do($db, sub {
if ($buildStatus == 0) {

my $size = 0;
my $closureSize = 0;
my $releaseName;

my @closure = computeFSClosure(0, 0, values %outputs);
foreach my $path (@closure) {
my ($deriver, $hash, $time, $narSize, $refs) = queryPathInfo($path, 0);
$closureSize += $narSize;
$size += $narSize if grep { $path eq $_ } values(%outputs);
}

foreach my $path (values %outputs) {
$buildStatus = 6 if $buildStatus == 0 && -f "$path/nix-support/failed";
$releaseName //= getReleaseName($path);
}

$build->update(
{ releasename => $releaseName
, size => $size
, closuresize => $closureSize
});

addBuildProducts($db, $build);
}

$build->update(
{ finished => 1
, busy => 0
, locker => ''
, logfile => ''
, iscachedbuild => $isCachedBuild
, buildstatus => $buildStatus
, starttime => $startTime
, stoptime => $stopTime // time()
, errormsg => $errormsg
});

});
}

my $buildId = $ARGV[0] or die "syntax: $0 BUILD-ID\n";
print STDERR "performing build $buildId\n";

# Lock the build. If necessary, steal the lock from the parent
# process (runner.pl). This is so that if the runner dies, the
# children (i.e. the build.pl instances) can continue to run and won't
# have the lock taken away.
my $build;
txn_do($db, sub {
$build = $db->resultset('Builds')->find($buildId);
die "build $buildId doesn't exist\n" unless defined $build;
die "build $buildId already done\n" if $build->finished;
if ($build->busy != 0 && $build->locker != getppid) {
die "build $buildId is already being built";
}
$build->update({busy => 1, locker => $$});
$build->buildsteps->search({busy => 1})->delete;
$build->buildproducts->delete;
});

die unless $build;

# Do the build. If it throws an error, unlock the build so that it
# can be retried.
eval {
doBuild $build;
print "done\n";
};
if ($@) {
warn $@;
txn_do($db, sub {
$build->update({busy => 0, locker => $$});
});
}
if ($ENV{'HYDRA_MAIL_TEST'}) {
my $build = $db->resultset('Builds')->find($buildId);
notifyBuildFinished(\@plugins, $build, []);
exit 0;
}

notifyBuildFinished(\@plugins, $build, $dependents);
# Mark any remaining active build steps as aborted.
$build->buildsteps->search({ busy => 1 })->update({ busy => 0, status => 4, stoptime => time });

elsif (/^@\s+substituter-failed\s+(\S+)\s+(\S+)\s+(\S+)$/) {
my $path = $1;
txn_do($db, sub {
my $step = $build->buildsteps->find({stepnr => $buildSteps{$path}}) or die;
$step->update({busy => 0, status => 1, errormsg => $3, stoptime => time});
});
}

else {
print STDERR "unknown Nix trace message: $_";
}
, starttime => time
$errormsg .= $_;

"--option print-missing false " .
if (!isValidPath($drvPath)) {
$buildStatus = 3;
$errormsg = "derivation was garbage-collected prior to build";
goto done;
}

my $dependents = [];
}

sub nextFreeStepNr {
my ($build) = @_;
my $max = $build->buildsteps->find(
{}, {select => {max => 'stepnr + 1'}, as => ['max']});
return (defined $max && defined $max->get_column('max')) ? $max->get_column('max') : 1;
use Nix::Store;
use Hydra::Plugin;
file deletion: hydra_queue_runner.pl (----------)Runner.pl (----------)hydra-queue-runner (---r------)hydra_queue_runner.pl (---r------)hydra_queue_runner.pl (---r------)runner.pl (----------)hydra_queue_runner.pl.in (---r------)

[24.1560]→[24.38:83](∅→∅),[24.83]→[24.7823:7823](∅→∅),[24.1]→[24.154:187](∅→∅),[24.187]→[24.7823:7823](∅→∅),[24.2543]→[24.485:527](∅→∅),[24.527]→[24.7823:7823](∅→∅),[24.1560]→[24.1973:2018](∅→∅),[24.2018]→[24.7823:7823](∅→∅),[24.2543]→[24.2792:2837](∅→∅),[24.2837]→[24.7823:7823](∅→∅),[24.4]→[24.9347:9380](∅→∅),[24.9380]→[24.7823:7823](∅→∅),[24.2543]→[24.2199118:2199166](∅→∅),[24.2199166]→[24.7823:7823](∅→∅)

#! /var/run/current-system/sw/bin/perl

use strict;
use Cwd;
use File::Basename;
use POSIX qw(dup2 :sys_wait_h);
use Hydra::Schema;
use Hydra::Helper::Nix;
use Hydra::Model::DB;
use IO::Handle;
use Nix::Store;
use Set::Scalar;

chdir Hydra::Model::DB::getHydraPath or die;
my $db = Hydra::Model::DB->new();

STDOUT->autoflush();

my $lastTime;

#$SIG{CHLD} = 'IGNORE';

sub unlockDeadBuilds {
# Unlock builds whose building process has died.
txn_do($db, sub {
my @builds = $db->resultset('Builds')->search({finished => 0, busy => 1});
foreach my $build (@builds) {
my $pid = $build->locker;
my $unlock = 0;
if ($pid == $$) {
if (!defined $lastTime || $build->starttime < $lastTime - 300) {
$unlock = 1;
}
} elsif (kill(0, $pid) != 1) { # see if we can signal the process
$unlock = 1;
}
if ($unlock) {
print "build ", $build->id, " pid $pid died, unlocking\n";
$build->update({ busy => 0, locker => "" });
$build->buildsteps->search({ busy => 1 })->update({ busy => 0, status => 4, stoptime => time });
}
}
});
}

sub findBuildDependencyInQueue {
my ($buildsByDrv, $build) = @_;
return undef unless isValidPath($build->drvpath);
my @deps = grep { /\.drv$/ && $_ ne $build->drvpath } computeFSClosure(0, 0, $build->drvpath);
return unless scalar @deps > 0;
foreach my $d (@deps) {
my $bs = $buildsByDrv->{$d};
next unless defined $bs;
return $db->resultset('Builds')->find((@$bs)[0]);
}
return undef;
}

sub checkBuilds {
# print "looking for runnable builds...\n";

my @buildsStarted;

my $machines = getMachines;

txn_do($db, sub {

# Cache scheduled builds by derivation path to speed up
# findBuildDependencyInQueue.
my $buildsByDrv = {};
push @{$buildsByDrv->{$_->drvpath}}, $_->id
foreach $db->resultset('Builds')->search({ finished => 0 });

# Builds in the queue of which a dependency is already building.
my $blockedBuilds = Set::Scalar->new();
blockBuilds($buildsByDrv, $blockedBuilds, $_)
foreach $db->resultset('Builds')->search({ finished => 0, busy => 1 });

# Get the system types for the runnable builds.
my @systemTypes = $db->resultset('Builds')->search(
{ finished => 0, busy => 0 },
{ join => ['project'], select => ['system'], as => ['system'], distinct => 1 });

# Get the total number of scheduling shares.
my $totalShares = getTotalShares($db) || 1;

# For each system type, select up to the maximum number of
# concurrent build for that system type.
foreach my $system (@systemTypes) {
# How many builds are already currently executing for this
# system type?
my $nrActive = $db->resultset('Builds')->search(
{finished => 0, busy => 1, system => $system->system})->count;

(my $systemTypeInfo) = $db->resultset('SystemTypes')->search({system => $system->system});
my $max = defined $systemTypeInfo ? $systemTypeInfo->maxconcurrent : $maxConcurrent{$system->system} // 2;

my $extraAllowed = $max - $nrActive;
next if $extraAllowed <= 0;

print STDERR "starting at most $extraAllowed builds for system ${\$system->system}\n";

j: while ($extraAllowed-- > 0) {

my @runnableJobsets = $db->resultset('Builds')->search(
{ finished => 0, busy => 0, system => $system->system },
{ select => ['project', 'jobset'], distinct => 1 });

next if @runnableJobsets == 0;

my $windowSize = 24 * 3600;
my $totalWindowSize = $windowSize * $max;

my @res;

foreach my $b (@runnableJobsets) {
my $jobset = $db->resultset('Jobsets')->find($b->get_column('project'), $b->get_column('jobset')) or die;

my $timeSpent = $timeSpentPerJobset->{$b->get_column('project')}->{$b->get_column('jobset')};

if (!defined $timeSpent) {
$timeSpent = $jobset->builds->search(
{ },
{ where => \ ("(finished = 0)")
, join => 'buildsteps'
, select => \ "sum(coalesce(buildsteps.stoptime, ${\time}) - buildsteps.starttime)"
, as => "sum" })->single->get_column("sum") // 0;

$timeSpent += $jobset->builds->search(
{ },
{ where => \ ("(me.stoptime >= " . (time() - $windowSize) . ")")
, join => 'buildsteps'
, select => \ "sum(coalesce(buildsteps.stoptime, ${\time}) - buildsteps.starttime)"
, as => "sum" })->single->get_column("sum") // 0;

# Add a 30s penalty for each started build. This
# is to account for jobsets that have running
# builds but no build steps yet.
$timeSpent += $jobset->builds->search({ finished => 0, busy => 1 })->count * $costPerBuild;

$timeSpentPerJobset->{$b->get_column('project')}->{$b->get_column('jobset')} = $timeSpent;
}

my $share = $jobset->schedulingshares || 1; # prevent division by zero
my $used = $timeSpent / ($totalWindowSize * ($share / $totalShares));

#printf STDERR "%s:%s: %d s, total used = %.2f%%, share used = %.2f%%\n", $jobset->get_column('project'), $jobset->name, $timeSpent, $timeSpent / $totalWindowSize * 100, $used * 100;

push @res, { jobset => $jobset, used => $used };
}

foreach my $r (sort { $a->{used} <=> $b->{used} } @res) {
my $jobset = $r->{jobset};
#print STDERR "selected ", $jobset->get_column('project'), ':', $jobset->name, "\n";

# Select the highest-priority build for this jobset.
my @builds = $jobset->builds->search(
{ finished => 0, busy => 0, system => $system->system },
{ order_by => ["priority DESC", "id"] });

foreach my $build (@builds) {
# Find a dependency of $build that has no queued
# dependencies itself. This isn't strictly necessary,
# but it ensures that Nix builds are done as part of
# their corresponding Hydra builds, rather than as a
# dependency of some other Hydra build.
while (my $dep = findBuildDependencyInQueue($buildsByDrv, $build)) {
$build = $dep;
}
next if $build->busy;

printf STDERR "starting build %d (%s:%s:%s) on %s; jobset at %.2f%% of its share\n",
$build->id, $build->project->name, $build->jobset->name, $build->job->name, $build->system, $r->{used} * 100;

my $logfile = getcwd . "/logs/" . $build->id;
mkdir(dirname $logfile);
unlink($logfile);
$build->update(
{ busy => 1
, locker => $$
, logfile => $logfile
});
push @buildsStarted, $build;
next j;
}
}

last; # nothing found, give up on this system type
}
}

$lastTime = time();

$_->update({ starttime => time() }) foreach @buildsStarted;
});

# Actually start the builds we just selected. We need to do this
# outside the transaction in case it aborts or something.
foreach my $build (@buildsStarted) {
my $id = $build->id;
eval {
my $logfile = $build->logfile;
my $child = fork();
die unless defined $child;
if ($child == 0) {
eval {
open LOG, ">$logfile" or die "cannot create logfile $logfile";
POSIX::dup2(fileno(LOG), 1) or die;
POSIX::dup2(fileno(LOG), 2) or die;
exec("hydra-build", $id);
};
warn "cannot start build $id: $@";
POSIX::_exit(1);
}
};
if ($@) {
warn $@;
txn_do($db, sub {
$build->update({ busy => 0, locker => $$ });
});
}
}

while (1) {
eval {
unlockDeadBuilds;

checkBuilds;
};
warn $@ if $@;

# print "sleeping...\n";
sleep(5);
}
# Clean up zombies.
while ((waitpid(-1, &WNOHANG)) > 0) { };

}

if (scalar(@ARGV) == 1 && $ARGV[0] eq "--unlock") {
unlockDeadBuilds;
exit 0;
}

$timeSpentPerJobset->{$jobset->get_column('project')}->{$jobset->name} += $costPerBuild;

blockBuilds($buildsByDrv, $blockedBuilds, $build);

next if $blockedBuilds->has($build->id);

my $costPerBuild = 30;

my $timeSpentPerJobset;
my %maxConcurrent;

foreach my $machineName (keys %{$machines}) {
foreach my $system (@{${$machines}{$machineName}{'systemTypes'}}) {
$maxConcurrent{$system} = (${$machines}{$machineName}{'maxJobs'} or 0) + ($maxConcurrent{$system} or 0)
}
}

sub blockBuilds {
my ($buildsByDrv, $blockedBuilds, $build) = @_;
my @rdeps = grep { /\.drv$/ && $_ ne $build->drvpath } computeFSClosure(1, 0, $build->drvpath);
foreach my $drv (@rdeps) {
my $bs = $buildsByDrv->{$drv};
next if !defined $bs;
$blockedBuilds->insert($_) foreach @$bs;
}
}

# Given a build, return an arbitrary queued build on which this build
# depends; or undef if no such build exists.
edit in src/script/Makefile.am at line 7

[24.422]→[24.422:442](∅→∅)

hydra-build \
edit in src/script/Makefile.am at line 8

[24.465]→[24.465:491](∅→∅)

hydra-queue-runner \
replacement in src/script/hydra-eval-guile-jobs.in at line 107

[17.438]→[17.438:504](∅→∅)

;; the time 'hydra-build' attempts to build it.

[17.438]

[17.504]

;; the time 'hydra-queue-runner' attempts to build it.