* Register GC roots properly.

Dependencies

• [2] S3ZLZP3N * Cut off builds after half an hour of apparent inactivity. This
• [3] BDSD2JLV * Speed up manifest generation.
• [4] SHZLOM5M * eval-jobs -> hydra_eval_jobs.
• [5] NEWDDAOF * Allow users to change the value of a build's "keep" flag, which
• [6] IN272KZW * Automatically keep all builds in the latest successful release in
• [7] H7CNGK4O * Log evaluation errors etc. in the DB.
• [8] WZ3AEJ67 * hydra_update_gc_roots.pl registers build outputs that should be kept
• [9] 4LWGZL33
• [10] CLJQCY2X * Store info about all the build actions and allow them to be
• [11] 37R34XJO * Negative caching: don't perform a build if a dependency already
• [12] AFTXA575 * $HYDRA_DATA environment variable.
• [13] TLZ2SPBR
• [14] 7YBYT2LQ
• [15] 2T42QGZD * Register builds as GC roots so they don't get deleted.
• [16] X27GNHDV * Basic job info in the database.
• [17] 4X6NS66Q * Keep the most recent builds for each job.
• [18] YGRLM2SK * Export all relevant info about the derivation.
• [19] POPU2ATH * hydra_scheduler: use eval-jobs.
• [20] BD3GRK4B * Get rid of "positive failures" and separate log phases.
• [*] 4N5APGRG * Start of a helper tool to evaluate job expressions efficiently.
• [*] KD5237CU * eval-jobs now efficiently evaluates all Hydra jobs from a
• [*] 2GK5DOU7 * Downloading closures.
• [*] N22GPKYT * Put info about logs / build products in the DB.

Change contents

• edit in src/c/hydra_eval_jobs.cc at line 20
  [6.73]

  [6.73]
  +
• edit in src/c/hydra_eval_jobs.cc at line 22
  [6.74]

  [6.74]
  + static Path gcRootsDir;
• edit in src/c/hydra_eval_jobs.cc at line 24
  [6.75]

  [6.75]
  +
• replacement in src/c/hydra_eval_jobs.cc at line 103
  [6.712]→[6.712:747](∅→∅)
  − Path outPath, drvPath;
  [6.712]

  [6.747]
  + Path drvPath;
• replacement in src/c/hydra_eval_jobs.cc at line 108
  [6.158]→[6.158:217](∅→∅)
  − xmlAttrs["drvPath"] = drv.queryDrvPath(state);
  [6.158]

  [6.217]
  + xmlAttrs["drvPath"] = drvPath = drv.queryDrvPath(state);
• edit in src/c/hydra_eval_jobs.cc at line 116
  [6.953]

  [6.1096]
  + /* Register the derivation as a GC root. !!! This

  + registers roots for jobs that we may have already

  + done. */

  + Path root = gcRootsDir + "/" + baseNameOf(drvPath);

  + if (!pathExists(root)) addPermRoot(drvPath, root, false);

  +
• edit in src/c/hydra_eval_jobs.cc at line 182
  [23.2867]

  [23.2867]
  + else if (arg == "--gc-roots-dir") {

  + if (i == args.end()) throw UsageError("missing argument");

  + gcRootsDir = *i++;

  + }
• edit in src/c/hydra_eval_jobs.cc at line 193
  [6.261]

  [6.261]
  +

  + if (gcRootsDir == "") throw UsageError("--gc-roots-dir not specified");
• replacement in src/lib/Hydra/Helper/Nix.pm at line 12
  [3.74]→[6.17:48](∅→∅),[6.675]→[6.17:48](∅→∅)
  − registerRoot getGCRootsDir
  [3.74]

  [6.675]
  + registerRoot getGCRootsDir gcRootFor
• replacement in src/lib/Hydra/Helper/Nix.pm at line 83
  [6.111]→[6.111:181](∅→∅)
  − return "/nix/var/nix/gcroots/per-user/$ENV{LOGNAME}/hydra-roots";
  [6.111]

  [6.181]
  + my $dir = "/nix/var/nix/gcroots/per-user/$ENV{LOGNAME}/hydra-roots";

  + mkpath $dir if !-e $dir;

  + return $dir;
• replacement in src/lib/Hydra/Helper/Nix.pm at line 89
  [6.185]→[6.185:204](∅→∅)
  − sub registerRoot {
  [6.185]

  [6.204]
  + sub gcRootFor {
• edit in src/lib/Hydra/Helper/Nix.pm at line 91
  [6.225]

  [6.225]
  + return getGCRootsDir . basename $path;

  + }
• replacement in src/lib/Hydra/Helper/Nix.pm at line 94
  [6.226]→[6.226:262](∅→∅)
  − my $gcRootsDir = getGCRootsDir;
  [6.226]

  [6.262]
  +

  + sub registerRoot {

  + my ($path) = @_;

  +

  + my $link = gcRootFor $path;
• edit in src/lib/Hydra/Helper/Nix.pm at line 100
  [6.267]→[6.267:369](∅→∅)
  − mkpath($gcRootsDir) if !-e $gcRootsDir;

  −

  − my $link = "$gcRootsDir/" . basename $path;

  −
• replacement in src/lib/Hydra/Helper/Nix.pm at line 102
  [6.420]→[6.420:488](∅→∅)
  − or die "cannot create symlink in $gcRootsDir to $path";
  [6.420]

  [6.488]
  + or die "cannot create GC root `$link' to `$path'";
• edit in src/script/hydra_build.pl at line 30
  [6.3483]→[6.495:527](∅→∅),[6.3622]→[6.495:527](∅→∅)
  −

  − registerRoot $outPath;
• replacement in src/script/hydra_build.pl at line 76
  [2.87]→[6.5065:5140](∅→∅),[6.5065]→[6.5065:5140](∅→∅)
  − "--log-type flat --print-build-trace --realise $drvPath 2>&1";
  [2.87]

  [6.5140]
  + "--log-type flat --print-build-trace --realise $drvPath " .

  + "--add-root " . gcRootFor $outPath . " 2>&1";
• edit in src/script/hydra_scheduler.pl at line 270
  [6.11037]→[6.1343:1352](∅→∅),[6.1352]→[6.529:642](∅→∅),[6.529]→[6.529:642](∅→∅)
  −

  − # !!! this should really by done by nix-instantiate to prevent a GC race.

  − registerRoot $drvPath;
• replacement in src/script/hydra_scheduler.pl at line 330
  [6.4086]→[4.619:687](∅→∅)
  − "hydra_eval_jobs", $nixExprPath, inputsToArgs($inputInfo));
  [6.4086]

  [6.4164]
  + "hydra_eval_jobs", $nixExprPath, "--gc-roots-dir", getGCRootsDir,

  + inputsToArgs($inputInfo));
• edit in src/script/hydra_scheduler.pl at line 333
  [6.4252]

  [6.7781]
  +

  + print STDERR "$stderr";
• replacement in src/script/hydra_update_gc_roots.pl at line 86
  [6.1500]→[6.689:769](∅→∅)
  − # For scheduled builds, we register the derivation and the output as a GC root.
  [6.1500]

  [5.2315]
  + # For scheduled builds, we register the derivation as a GC root.
• replacement in src/script/hydra_update_gc_roots.pl at line 93
  [6.1863]→[6.911:949](∅→∅)
  − registerRoot $build->outpath;
  [6.1863]

  [6.1863]
  + registerRoot $build->outpath if -e $build->outpath;