Start of single-process hydra-queue-runner

[?]

May 28, 2015, 3:39 PM

24BMQDZAWDQ7VNIA7TIROXSOYLOJBNZ2E4264WHWNJAEN6ZB3UOAC

Dependencies

[2] D4QVCQUS Use hashFile instead of nix-hash
[3] JGLE5BRN Add separate build step status codes for cached failures and timeouts
[4] HZDVCFCB Don't add a nix-build build product unless $out is a directory
[5] KNLKTCDM Use pkgconfig to find Nix
[6] DVLDHOUZ hydra-evaluator: Reduce verbosity
[7] JYZEGF56 Create Builds with iscurrent set
[8] UN2KZL3A Rename c -> hydra-eval-jobs
[9] RBQEBVT5 Doh
[10] PMNWRTGJ Add multiple output support
[11] 6ZB4CIW6 Security: Improve checking of build products
[12] NDJ6PZB7 * Fix symbol to look for.
[13] FSZJXAHR add sqlite to buildinputs
[14] YDVFPMKP Security: Ensure that a build product refers to the Nix store
[15] 3XTHEUMP * Implemented the clone feature.
[16] AKRVETP5 Handle UTF-8 characters in eval error messages
[17] YQWH4POV * Simplify.
[18] R3ON2RJ3 hydra: missing argument
[19] UHMUHQMU hydra: fix tarball build, add pre suffix to tarballs
[20] JZE7DC2F Whitespace
[21] TOTSL2RB When checking if build is already built, check for potential 'failed with result'. Fixes issue #7.
[22] RXVJFQ5A Evaluator cleanups
[23] M3A5PZIH hydra: Clarify the dependency on BDW-GC.
[24] SRBU2RAE Warn against multiple jobs with the same name
[25] Y6AHH4TH Remove the logfile and logSize columns from the database
[26] FTPCV25M Store aggregate members in the database
[27] X5QBLQDG
[28] WSCHBYEN add topgit to hydra's path
[29] 6L3ZM55S Add font for the captcha
[30] XCHBOA3S svn -> subversion
[31] FM4O2L4M hydra: if evaluator sees cached build, also add the buildproducts
[32] U2BNO3C5 hydra: fix build job
[33] JAH3UPWA Support revision control systems via plugins
[34] PHNLYPKB Call buildFinished when a cached build is added
[35] BIVZGPUT Optimise clickable rows
[36] HJECG75O hydrA: add some pkgs to buildinputs for tests
[37] 2WRTOU2Z Cleanup
[38] A22P7HCO hydra: at evaluation, check if path is already built, and mark as built in stead of adding to the queue.
[39] XDKFASXT Add bzip2 to buildInputs.
[40] 2GUAKGTB Fix indentation of build.tt
[41] INNOEHO6 * Fix getBuildLog for bzip2'd files.
[42] EDRUQ4UK Die TABs die
[43] T4LLYESZ * Nix expression for building Hydra.
[44] QEC5HR4R Always record inputs passed through -I in the BuildInputs table
[45] LOHWNXEJ Cleanup
[46] 6GEU36HW Remove obsolete CSS classes
[47] IJSJLRZH Disallow build products that are symlinks
[48] PY5GVGC7 Implemented quoted strings support in hydra-build-products to allow file names with spaces + testcase
[49] 6K5PBUUN Use buildEnv to combine Hydra's Perl dependencies
[50] PH3DFCNU Render machine correctly if it doesn't contain @
[51] L7RW467Z Add a test for darcs inputs.
[52] BMSQD2ZH Indentation
[53] CQTN62OH Die tabs die
[54] JOYONH2K Prevent multiple builds with the same (job, outPath) tuple from being added
[55] VM3TQZ65 * hydra: make manual.pdf
[56] QW7GTTXI Don't maintain BuildInputs anymore
[57] YFPZ46YK * hydra: added variant of build input type, 'build output (same system)' to allow better continous integration in one jobset for multiple system. it makes sure that the system of the build that is passed as input for a job has the same system as the job.
[58] HPEG2RHV Merge the BuildResultInfo table into the Builds table
[59] VOPKXPFE Add OpenSSL as a dependency because Nix needs it
[60] OOQ2D3KC * Refactoring: move fetchInput out of hydra_scheduler into a separate
[61] IMQRX4MP hydra-eval-jobs: Use JSON instead of XML
[*] FV2M6MOT hydra: use autoconf/-make
[*] CLXEECMF * Start putting build results in a database.
[*] J5UVLXOK * Start of a basic Catalyst web interface.
[*] N22GPKYT * Put info about logs / build products in the DB.
[*] L2E6EVE2 * Merged the Build and Job tables.
[*] 3PNG7NIB Remove trailing whitespace
[*] R6B5CAFF Let Builds.timestamp refer to the time the build was added
[*] CLJQCY2X * Store info about all the build actions and allow them to be

Change contents

edit in configure.ac at line 76

[9.69]

[63.16483]

src/hydra-queue-runner/Makefile
replacement in release.nix at line 132

[10.1780]→[5.213:281](∅→∅)

[ makeWrapper libtool unzip nukeReferences pkgconfig sqlite

[10.1780]

[10.0]

[ makeWrapper libtool unzip nukeReferences pkgconfig sqlite libpqxx
replacement in src/Makefile.am at line 1

[10.21]→[8.0:54](∅→∅)

SUBDIRS = hydra-eval-jobs sql script lib root xsl ttf

[10.21]

[10.62]

SUBDIRS = hydra-eval-jobs hydra-queue-runner sql script lib root xsl ttf
file addition: hydra-queue-runner (d--r------)

[64.4]
file addition: Makefile.am (----------)

[0.187]

bin_PROGRAMS = hydra-queue-runner

hydra_queue_runner_SOURCES = hydra-queue-runner.cc build-result.cc
hydra_queue_runner_LDADD = $(NIX_LIBS) -lpqxx

AM_CXXFLAGS = $(NIX_CFLAGS) -Wall
file addition: build-result.cc (----------)

[0.187]

#include "build-result.hh"
#include "store-api.hh"
#include "misc.hh"
#include "util.hh"

using namespace nix;

BuildResult getBuildResult(const Derivation & drv)
{
BuildResult res;

/* Compute the closure size. */
PathSet outputs;
for (auto & output : drv.outputs)
outputs.insert(output.second.path);
PathSet closure;
for (auto & output : outputs)
computeFSClosure(*store, output, closure);
for (auto & path : closure) {
auto info = store->queryPathInfo(path);
res.closureSize += info.narSize;
if (outputs.find(path) != outputs.end()) res.size += info.narSize;
}

/* Get build products. */
bool explicitProducts = false;

for (auto & output : outputs) {
Path productsFile = output + "/nix-support/hydra-build-products";
if (!pathExists(productsFile)) continue;
explicitProducts = true;

/* For security, resolve symlinks. */
productsFile = canonPath(productsFile, true);
if (!isInStore(productsFile)) continue;

// FIXME: handle I/O errors

auto contents = readFile(productsFile);
auto lines = tokenizeString<Strings>(contents, "\n");

for (auto & line : lines) {
BuildProduct product;

auto words = tokenizeString<Strings>(line);
if (words.size() < 3) continue;
product.type = words.front(); words.pop_front();
product.subtype = words.front(); words.pop_front();
if (string(words.front(), 0, 1) == "\"") {
// FIXME:
throw Error("FIXME");
} else {
product.path = words.front(); words.pop_front();
}
product.defaultPath = words.empty() ? "" : words.front();

/* Ensure that the path exists and points into the
Nix store. */
if (product.path == "" || product.path[0] != '/') continue;
product.path = canonPath(product.path, true);
if (!isInStore(product.path) || !pathExists(product.path)) continue;

/* FIXME: check that the path is in the input closure
of the build? */

product.name = product.path == output ? "" : baseNameOf(product.path);

struct stat st;
if (stat(product.path.c_str(), &st))
throw SysError(format("getting status of ‘%1%’") % product.path);

if (S_ISREG(st.st_mode)) {
product.isRegular = true;
product.fileSize = st.st_size;
product.sha1hash = hashFile(htSHA1, product.path);
product.sha256hash = hashFile(htSHA256, product.path);
}

res.products.push_back(product);
}
}

/* If no build products were explicitly declared, then add all
outputs as a product of type "nix-build". */
if (!explicitProducts) {
for (auto & output : drv.outputs) {
BuildProduct product;
product.path = output.second.path;
product.type = "nix-build";
product.subtype = output.first == "out" ? "" : output.first;
product.name = storePathToName(product.path);

struct stat st;
if (stat(product.path.c_str(), &st))
throw SysError(format("getting status of ‘%1%’") % product.path);
if (S_ISDIR(st.st_mode))
res.products.push_back(product);
}
}

/* Get the release name from $output/nix-support/hydra-release-name. */
for (auto & output : outputs) {
Path p = output + "/nix-support/hydra-release-name";
if (!pathExists(p)) continue;
// FIXME: handle I/O error
res.releaseName = trim(readFile(p));
// FIXME: validate release name
}

return res;
}
file addition: build-result.hh (----------)

[0.187]

#pragma once

#include "hash.hh"
#include "derivations.hh"

struct BuildProduct
{
nix::Path path, defaultPath;
std::string type, subtype, name;
bool isRegular = false;
nix::Hash sha1hash, sha256hash;
off_t fileSize = 0;
BuildProduct() { }
};

struct BuildResult
{
std::string releaseName;

unsigned long long closureSize = 0, size = 0;

std::list<BuildProduct> products;
};

BuildResult getBuildResult(const nix::Derivation & drv);
file addition: hydra-queue-runner.cc (----------)

[0.187]

#include <iostream>
#include <memory>
#include <map>
#include <pqxx/pqxx>

#include "build-result.hh"
#include "store-api.hh"
#include "derivations.hh"
#include "shared.hh"
#include "globals.hh"

using namespace nix;

typedef enum {
bsSuccess = 0,
bsFailed = 1,
bsDepFailed = 2,
bsAborted = 3,
bsFailedWithOutput = 6,
} BuildStatus;

typedef enum {
bssSuccess = 0,
bssFailed = 1,
bssAborted = 4,
bssBusy = 100, // not stored
} BuildStepStatus;

struct Connection : pqxx::connection
{
Connection() : pqxx::connection("dbname=hydra") { };
};

typedef unsigned int BuildID;

struct Build
{
typedef std::shared_ptr<Build> ptr;
typedef std::weak_ptr<Build> wptr;

BuildID id;
Path drvPath;
std::map<string, Path> outputs;

bool finishedInDB;

Build() : finishedInDB(false) { }
};

struct Step
{
typedef std::shared_ptr<Step> ptr;
typedef std::weak_ptr<Step> wptr;
Path drvPath;
Derivation drv;

/* The build steps on which this step depends. */
std::set<Step::ptr> deps;

/* The build steps that depend on this step. */
std::vector<Step::wptr> rdeps;

/* Builds that have this step as the top-level derivation. */
std::vector<Build::wptr> builds;
};

class State
{
private:
/* The queued builds. */
std::map<BuildID, Build::ptr> builds;

/* All active or pending build steps (i.e. dependencies of the
queued builds). */
std::map<Path, Step::ptr> steps;

/* Build steps that have no unbuilt dependencies. */
std::set<Step::ptr> runnable;

public:
State();

~State();

void markActiveBuildStepsAsAborted(pqxx::connection & conn, time_t stopTime);

int createBuildStep(pqxx::work & txn, time_t startTime, Build::ptr build, Step::ptr step,
BuildStepStatus status, const std::string & errorMsg = "", BuildID propagatedFrom = 0);

void finishBuildStep(pqxx::work & txn, time_t stopTime, BuildID buildId, int stepNr,
BuildStepStatus status, const string & errorMsg = "", BuildID propagatedFrom = 0);

void updateBuild(pqxx::work & txn, Build::ptr build, BuildStatus status);

void getQueuedBuilds(pqxx::connection & conn);

Step::ptr createStep(const Path & drvPath);

void destroyStep(Step::ptr step, bool proceed);

/* Get the builds that depend on the given step. */
std::set<Build::ptr> getDependentBuilds(Step::ptr step);

void doBuildSteps();

void doBuildStep(Step::ptr step);

void markSucceededBuild(pqxx::work & txn, Build::ptr build,
const BuildResult & res, bool isCachedBuild, time_t startTime, time_t stopTime);
};

State::State()
{
}

State::~State()
{
try {
Connection conn;
printMsg(lvlError, "clearing active build steps...");
markActiveBuildStepsAsAborted(conn, time(0));
} catch (...) {
ignoreException();
}
}

void State::markActiveBuildStepsAsAborted(pqxx::connection & conn, time_t stopTime)
{
pqxx::work txn(conn);
auto stm = txn.parameterized
("update BuildSteps set busy = 0, status = $1, stopTime = $2 where busy = 1")
((int) bssAborted);
if (stopTime) stm(stopTime); else stm();
stm.exec();
txn.commit();
}

int State::createBuildStep(pqxx::work & txn, time_t startTime, Build::ptr build, Step::ptr step,
BuildStepStatus status, const std::string & errorMsg, BuildID propagatedFrom)
{
auto res = txn.parameterized("select max(stepnr) from BuildSteps where build = $1")(build->id).exec();
int stepNr = res[0][0].is_null() ? 1 : res[0][0].as<int>() + 1;

auto stm = txn.parameterized
("insert into BuildSteps (build, stepnr, type, drvPath, busy, startTime, system, status, propagatedFrom, errorMsg, stopTime) values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)")
(build->id)(stepNr)(0)(step->drvPath)(status == bssBusy ? 1 : 0)(startTime)(step->drv.platform);
if (status == bssBusy) stm(); else stm((int) status);
if (propagatedFrom) stm(propagatedFrom); else stm();
if (errorMsg != "") stm(errorMsg); else stm();
if (status == bssBusy) stm(); else stm(startTime);
stm.exec();

for (auto & output : step->drv.outputs)
txn.parameterized
("insert into BuildStepOutputs (build, stepnr, name, path) values ($1, $2, $3, $4)")
(build->id)(stepNr)(output.first)(output.second.path).exec();

return stepNr;
}

void State::finishBuildStep(pqxx::work & txn, time_t stopTime, BuildID buildId, int stepNr,
BuildStepStatus status, const std::string & errorMsg, BuildID propagatedFrom)
{
auto stm = txn.parameterized
("update BuildSteps set busy = 0, status = $1, propagatedFrom = $4, errorMsg = $5, stopTime = $6 where build = $2 and stepnr = $3")
((int) status)(buildId)(stepNr);
if (propagatedFrom) stm(propagatedFrom); else stm();
if (errorMsg != "") stm(errorMsg); else stm();
if (stopTime) stm(stopTime); else stm();
stm.exec();
}

void State::getQueuedBuilds(pqxx::connection & conn)
{
pqxx::work txn(conn);

// FIXME: query only builds with ID higher than the previous
// highest.
auto res = txn.exec("select * from Builds where finished = 0");

// FIXME: don't process inside a txn.
for (auto const & row : res) {
BuildID id = row["id"].as<BuildID>();
if (builds.find(id) != builds.end()) continue;

Build::ptr build(new Build);
build->id = id;
build->drvPath = row["drvPath"].as<string>();

printMsg(lvlInfo, format("loading build %1% (%2%:%3%:%4%)") % id % row["project"] % row["jobset"] % row["job"]);

if (!store->isValidPath(build->drvPath)) {
/* Derivation has been GC'ed prematurely. */
Connection conn;
pqxx::work txn(conn);
txn.parameterized
("update Builds set finished = 1, buildStatus = $2, startTime = $3, stopTime = $3, errorMsg = $4 where id = $1")
(build->id)
((int) bsAborted)
(time(0))
("derivation was garbage-collected prior to build").exec();
txn.commit();
continue;
}

Step::ptr step = createStep(build->drvPath);
if (!step) {
Derivation drv = readDerivation(build->drvPath);
BuildResult res = getBuildResult(drv);

Connection conn;
pqxx::work txn(conn);
time_t now = time(0);
markSucceededBuild(txn, build, res, true, now, now);
txn.commit();

continue;
}

step->builds.push_back(build);

builds[id] = build;
}
}

Step::ptr State::createStep(const Path & drvPath)
{
auto prev = steps.find(drvPath);
if (prev != steps.end()) return prev->second;

printMsg(lvlInfo, format("considering derivation ‘%1%’") % drvPath);

Step::ptr step(new Step);
step->drvPath = drvPath;
step->drv = readDerivation(drvPath);

/* Are all outputs valid? */
bool valid = true;
for (auto & i : step->drv.outputs) {
if (!store->isValidPath(i.second.path)) {
valid = false;
break;
}
}

// FIXME: check whether all outputs are in the binary cache.
if (valid) return 0;

/* No, we need to build. */
printMsg(lvlInfo, format("creating build step ‘%1%’") % drvPath);

/* Create steps for the dependencies. */
for (auto & i : step->drv.inputDrvs) {
Step::ptr dep = createStep(i.first);
if (dep) {
step->deps.insert(dep);
dep->rdeps.push_back(step);
}
}

steps[drvPath] = step;

if (step->deps.empty()) runnable.insert(step);

return step;
}

void State::destroyStep(Step::ptr step, bool proceed)
{
steps.erase(step->drvPath);

for (auto & rdep_ : step->rdeps) {
auto rdep = rdep_.lock();
if (!rdep) continue;
assert(rdep->deps.find(step) != rdep->deps.end());
rdep->deps.erase(step);
if (proceed) {
/* If this rdep has no other dependencies, then we can now
build it. */
if (rdep->deps.empty())
runnable.insert(rdep);
} else
/* If ‘step’ failed, then delete all dependent steps as
well. */
destroyStep(rdep, false);
}

for (auto & build_ : step->builds) {
auto build = build_.lock();
if (!build) continue;
assert(build->drvPath == step->drvPath);
assert(build->finishedInDB);
}
}

std::set<Build::ptr> State::getDependentBuilds(Step::ptr step)
{
std::set<Step::ptr> done;
std::set<Build::ptr> res;

std::function<void(Step::ptr)> visit;

visit = [&](Step::ptr step) {
if (done.find(step) != done.end()) return;
done.insert(step);

for (auto & build : step->builds) {
auto build2 = build.lock();
if (build2) res.insert(build2);
}

for (auto & rdep : step->rdeps) {
auto rdep2 = rdep.lock();
if (rdep2) visit(rdep2);
}
};

visit(step);

return res;
}

void State::doBuildSteps()
{
while (!runnable.empty()) {
printMsg(lvlInfo, format("%1% runnable steps") % runnable.size());
Step::ptr step = *runnable.begin();
runnable.erase(step);
doBuildStep(step);
}
}

void State::doBuildStep(Step::ptr step)
{
assert(step->deps.empty());

/* There can be any number of builds in the database that depend
on this derivation. Arbitrarily pick one (though preferring
those build of which this is the top-level derivation) for the
purpose of creating build steps. We could create a build step
record for every build, but that could be very expensive
(e.g. a stdenv derivation can be a dependency of tens of
thousands of builds), so we don't. */
Build::ptr build;

auto builds = getDependentBuilds(step);

if (builds.empty()) {
/* Apparently all builds that depend on this derivation are
gone (e.g. cancelled). So don't bother. */
printMsg(lvlInfo, format("cancelling build step ‘%1%’") % step->drvPath);
destroyStep(step, true);
return;
}

for (auto build2 : builds)
if (build2->drvPath == step->drvPath) { build = build2; break; }

if (!build) build = *builds.begin();

printMsg(lvlInfo, format("performing build step ‘%1%’ (needed by %2% builds)") % step->drvPath % builds.size());

/* Create a build step record indicating that we started
building. */
Connection conn;
time_t startTime = time(0);
int stepNr;
{
pqxx::work txn(conn);
stepNr = createBuildStep(txn, startTime, build, step, bssBusy);
txn.commit();
}

bool success = false;
std::string errorMsg;
try {
store->buildPaths(PathSet({step->drvPath}));
success = true;
} catch (Error & e) {
errorMsg = e.msg();
}

time_t stopTime = time(0);

BuildResult res;
if (success) res = getBuildResult(step->drv);

// FIXME: handle failed-with-output

// FIXME: handle new builds having been added in the meantime.

{
pqxx::work txn(conn);

if (success) {

finishBuildStep(txn, stopTime, build->id, stepNr, bssSuccess);

/* Mark all builds of which this derivation is the top
level as succeeded. */
for (auto build2_ : step->builds) {
auto build2 = build2_.lock();
if (!build2) continue;
markSucceededBuild(txn, build2, res, false, startTime, stopTime);
}

} else {
/* Create failed build steps for every build that depends
on this. */
finishBuildStep(txn, stopTime, build->id, stepNr, bssFailed, errorMsg);

for (auto build2 : builds) {
if (build == build2) continue;
createBuildStep(txn, stopTime, build2, step, bssFailed, errorMsg, build->id);
}

/* Mark all builds that depend on this derivation as failed. */
for (auto build2 : builds) {
txn.parameterized
("update Builds set finished = 1, isCachedBuild = 0, buildStatus = $2, startTime = $3, stopTime = $4 where id = $1")
(build2->id)
((int) (build2->drvPath == step->drvPath ? bsFailed : bsDepFailed))
(startTime)
(stopTime).exec();
build2->finishedInDB = true; // FIXME: txn might fail
}
}

txn.commit();

}

/* Remove the build step from the graph. */
destroyStep(step, success);
}

void State::markSucceededBuild(pqxx::work & txn, Build::ptr build,
const BuildResult & res, bool isCachedBuild, time_t startTime, time_t stopTime)
{
auto stm = txn.parameterized
("update Builds set finished = 1, buildStatus = $2, startTime = $3, stopTime = $4, size = $5, closureSize = $6, releaseName = $7, isCachedBuild = $8 where id = $1")
(build->id)
((int) bsSuccess)
(startTime)
(stopTime)
(res.size)
(res.closureSize);
if (res.releaseName != "") stm(res.releaseName); else stm();
stm(isCachedBuild ? 1 : 0);
stm.exec();

unsigned int productNr = 1;
for (auto & product : res.products) {
auto stm = txn.parameterized
("insert into BuildProducts (build, productnr, type, subtype, fileSize, sha1hash, sha256hash, path, name, defaultPath) values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)")
(build->id)
(productNr++)
(product.type)
(product.subtype);
if (product.isRegular) stm(product.fileSize); else stm();
if (product.isRegular) stm(printHash(product.sha1hash)); else stm();
if (product.isRegular) stm(printHash(product.sha256hash)); else stm();
stm
(product.path)
(product.name)
(product.defaultPath).exec();
}

build->finishedInDB = true; // FIXME: txn might fail
}

int main(int argc, char * * argv)
{
return handleExceptions(argv[0], [&]() {
initNix();

settings.buildVerbosity = lvlVomit;
settings.useSubstitutes = false;

store = openStore();

/* FIXME: need some locking to prevent multiple instances of
hydra-queue-runner. */

Connection conn;

State state;

state.markActiveBuildStepsAsAborted(conn, 0);

state.getQueuedBuilds(conn);

state.doBuildSteps();
});
}
replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 25

[10.624]→[10.624:673](∅→∅),[10.673]→[10.67:89](∅→∅),[10.844]→[10.67:89](∅→∅),[10.156]→[10.67:89](∅→∅)

getReleaseName addBuildProducts restartBuild
getPrevJobsetEval

[10.624]

[10.156]

restartBuild getPrevJobsetEval
edit in src/lib/Hydra/Helper/AddBuilds.pm at line 29

[10.3176]→[10.99:120](∅→∅),[10.120]→[10.0:24](∅→∅),[10.24]→[10.4500:4670](∅→∅),[10.4670]→[10.374:401](∅→∅),[10.374]→[10.374:401](∅→∅),[10.401]→[10.3176:3177](∅→∅),[10.3176]→[10.3176:3177](∅→∅),[10.3177]→[10.2:3](∅→∅)

sub getReleaseName {
my ($outPath) = @_;
return undef unless -f "$outPath/nix-support/hydra-release-name";
my $releaseName = read_file("$outPath/nix-support/hydra-release-name");
chomp $releaseName;
return $releaseName;
}
edit in src/lib/Hydra/Helper/AddBuilds.pm at line 345

[10.348]→[10.3428:3430](∅→∅),[10.2233]→[10.3428:3430](∅→∅),[10.3428]→[10.3428:3430](∅→∅),[10.3430]→[10.0:1](∅→∅),[10.1]→[10.23:24](∅→∅),[10.24]→[10.1:52](∅→∅),[10.1]→[10.1:52](∅→∅),[10.87]→[10.87:110](∅→∅),[10.110]→[10.5122:5152](∅→∅),[10.5152]→[10.35:84](∅→∅),[10.84]→[10.110:111](∅→∅),[10.5152]→[10.110:111](∅→∅),[10.110]→[10.110:111](∅→∅),[10.111]→[10.5153:5445](∅→∅),[10.5445]→[10.0:78](∅→∅),[10.78]→[10.5515:5600](∅→∅),[10.5515]→[10.5515:5600](∅→∅),[10.5600]→[10.255:333](∅→∅),[10.139]→[10.5631:5669](∅→∅),[10.180]→[10.5631:5669](∅→∅),[10.333]→[10.5631:5669](∅→∅),[10.5631]→[10.5631:5669](∅→∅),[10.5669]→[10.140:288](∅→∅),[10.288]→[10.334:452](∅→∅),[10.146]→[10.5669:5707](∅→∅),[10.345]→[10.5669:5707](∅→∅),[10.452]→[10.5669:5707](∅→∅),[10.5669]→[10.5669:5707](∅→∅),[10.181]→[10.424:526](∅→∅),[10.424]→[10.424:526](∅→∅),[10.508]→[10.567:568](∅→∅),[10.526]→[10.567:568](∅→∅),[10.1982]→[10.567:568](∅→∅),[10.5707]→[10.567:568](∅→∅),[10.567]→[10.567:568](∅→∅),[10.568]→[10.5708:5760](∅→∅),[10.550]→[10.624:625](∅→∅),[10.2031]→[10.624:625](∅→∅),[10.5760]→[10.624:625](∅→∅),[10.624]→[10.624:625](∅→∅),[10.3431]→[10.5813:5961](∅→∅),[10.5961]→[2.0:116](∅→∅),[2.116]→[10.6277:6295](∅→∅),[10.6277]→[10.6277:6295](∅→∅),[10.910]→[10.3431:3432](∅→∅),[10.1247]→[10.3431:3432](∅→∅),[10.2538]→[10.3431:3432](∅→∅),[10.6295]→[10.3431:3432](∅→∅),[10.3431]→[10.3431:3432](∅→∅),[10.3432]→[10.6296:6364](∅→∅),[10.6364]→[10.421:422](∅→∅),[10.2603]→[10.421:422](∅→∅),[10.422]→[10.6365:6902](∅→∅),[10.6902]→[10.3055:3065](∅→∅),[10.3055]→[10.3055:3065](∅→∅),[10.3085]→[10.1275:1281](∅→∅),[10.1275]→[10.1275:1281](∅→∅),[10.1281]→[10.6903:6937](∅→∅),[10.1281]→[10.1948:1949](∅→∅),[10.6937]→[10.1948:1949](∅→∅),[10.1948]→[10.1948:1949](∅→∅),[10.1949]→[10.6938:7028](∅→∅),[10.7028]→[4.0:33](∅→∅),[4.33]→[10.3086:3243](∅→∅),[10.7028]→[10.3086:3243](∅→∅),[10.1293]→[10.3086:3243](∅→∅),[10.3243]→[10.7029:7098](∅→∅),[10.7098]→[10.3271:3356](∅→∅),[10.3271]→[10.3271:3356](∅→∅),[10.3356]→[10.1507:1513](∅→∅),[10.1507]→[10.1507:1513](∅→∅)

}

sub addBuildProducts {
my ($db, $build) = @_;

my $productnr = 1;
my $explicitProducts = 0;
my $storeDir = $Nix::Config::storeDir . "/";

foreach my $output ($build->buildoutputs->all) {
my $outPath = $output->path;
if (-e "$outPath/nix-support/hydra-build-products") {
$explicitProducts = 1;

open LIST, "$outPath/nix-support/hydra-build-products" or die;
while (<LIST>) {
/^([\w\-]+)\s+([\w\-]+)\s+("[^"]*"|\S+)(\s+(\S+))?$/ or next;
my $type = $1;
my $subtype = $2 eq "none" ? "" : $2;
my $path = substr($3, 0, 1) eq "\"" ? substr($3, 1, -1) : $3;
my $defaultPath = $5;

# Ensure that the path exists and points into the Nix store.
next unless File::Spec->file_name_is_absolute($path);
$path = pathIsInsidePrefix($path, $Nix::Config::storeDir);
next unless defined $path;
next unless -e $path;

# FIXME: check that the path is in the input closure
# of the build?

my $fileSize, my $sha1, my $sha256;

if (-f $path) {
my $st = stat($path) or die "cannot stat $path: $!";
$fileSize = $st->size;
$sha1 = hashFile("sha1", 0, $path);
$sha256 = hashFile("sha256", 0, $path);
}

my $name = $path eq $outPath ? "" : basename $path;

$db->resultset('BuildProducts')->create(
{ build => $build->id
, productnr => $productnr++
, type => $type
, subtype => $subtype
, path => $path
, filesize => $fileSize
, sha1hash => $sha1
, sha256hash => $sha256
, name => $name
, defaultpath => $defaultPath
});
}
close LIST;
}
}

return if $explicitProducts;

foreach my $output ($build->buildoutputs->all) {
my $outPath = $output->path;
next unless -d $outPath;
$db->resultset('BuildProducts')->create(
{ build => $build->id
, productnr => $productnr++
, type => "nix-build"
, subtype => $output->name eq "out" ? "" : $output->name
, path => $outPath
, name => $build->nixname
});
}
edit in src/lib/Hydra/Helper/AddBuilds.pm at line 420

[10.581]→[10.8248:8336](∅→∅),[10.8336]→[10.881:905](∅→∅),[10.5211]→[10.881:905](∅→∅),[10.905]→[10.8337:8455](∅→∅),[10.8455]→[10.2363:2418](∅→∅),[10.2418]→[10.8517:8909](∅→∅),[10.8517]→[10.8517:8909](∅→∅),[10.8909]→[10.942:968](∅→∅),[10.942]→[10.942:968](∅→∅),[10.968]→[10.582:614](∅→∅),[10.614]→[10.1001:1038](∅→∅),[10.1001]→[10.1001:1038](∅→∅),[10.1038]→[10.8910:8956](∅→∅),[10.8956]→[10.615:688](∅→∅),[10.74]→[10.615:688](∅→∅),[10.688]→[10.8957:9003](∅→∅),[10.9003]→[10.1290:1381](∅→∅),[10.1290]→[10.1290:1381](∅→∅),[10.1381]→[10.689:690](∅→∅)

# Are the outputs already in the Nix store? Then add a cached
# build.
my %extraFlags;
my $allValid = 1;
my $buildStatus;
my $releaseName;
foreach my $name (@outputNames) {
my $path = $buildInfo->{outputs}->{$name};
if (isValidPath($path)) {
if (-f "$path/nix-support/failed") {
$buildStatus = 6;
} else {
$buildStatus //= 0;
}
$releaseName //= getReleaseName($path);
} else {
$allValid = 0;
last;
}
}

if ($allValid) {
%extraFlags =
( finished => 1
, iscachedbuild => 1
, buildstatus => $buildStatus
, starttime => $time
, stoptime => $time
, releasename => $releaseName
);
} else {
%extraFlags = ( finished => 0 );
}
edit in src/lib/Hydra/Helper/AddBuilds.pm at line 435

[10.2478]

[10.4587]

, finished => 0
edit in src/lib/Hydra/Helper/AddBuilds.pm at line 439

[7.29]→[10.4638:4664](∅→∅),[10.4638]→[10.4638:4664](∅→∅)

, %extraFlags
replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 447

[10.726]→[10.1543:1580](∅→∅),[10.6016]→[10.1543:1580](∅→∅),[10.1580]→[6.169:244](∅→∅),[6.244]→[10.0:43](∅→∅),[10.1654]→[10.0:43](∅→∅),[10.987]→[10.0:43](∅→∅),[10.43]→[10.249:304](∅→∅),[10.43]→[10.987:1004](∅→∅),[10.304]→[10.987:1004](∅→∅),[10.2358]→[10.987:1004](∅→∅),[10.987]→[10.987:1004](∅→∅),[10.1004]→[6.245:358](∅→∅),[6.358]→[10.6419:6429](∅→∅),[10.6419]→[10.6419:6429](∅→∅)

if ($build->iscachedbuild) {
#print STDERR " marked as cached build ", $build->id, "\n";
addBuildProducts($db, $build);
notifyBuildFinished($plugins, $build, []);
} else {
print STDERR "added build ${\$build->id} (${\$jobset->project->name}:${\$jobset->name}:$jobName)\n";
}

[10.726]

[10.6965]

print STDERR "added build ${\$build->id} (${\$jobset->project->name}:${\$jobset->name}:$jobName)\n";
replacement in src/root/build.tt at line 36

[10.578]→[10.578:663](∅→∅)

INCLUDE renderDuration duration = step.stoptime - step.starttime;

[10.578]

[10.663]

IF step.stoptime;
INCLUDE renderDuration duration = step.stoptime - step.starttime;
ELSE;
%]?[%
END;
edit in src/root/build.tt at line 59

[3.197]

[10.1245]

[% ELSIF step.errormsg %]
Failed: [% HTML.escape(step.errormsg) %]
replacement in src/root/build.tt at line 62

[10.1270]→[10.1270:1354](∅→∅)

Failed: [% HTML.escape(step.errormsg) %]

[10.1270]

[10.1354]

Failed
edit in src/sql/hydra.sql at line 162

[67.9516]

[67.9516]

-- FIXME: remove (obsolete with the new queue runner)
edit in src/sql/hydra.sql at line 168

[68.37616]

[69.2348]

-- FIXME: remove startTime?
edit in src/sql/hydra.sql at line 212

[10.19973]

[70.7124]

-- TODO: normalize this. Currently there can be multiple BuildSteps
-- for a single step.