Remove hydra-eval-guile-jobs

[?]

Feb 15, 2020, 2:12 PM

QGLBQ5IOVSMBQRXXHO4LOJG4A7V3HSUZESCP2N235A7L4VVOBKCQC

Dependencies

[2] SVDJSA6U hydra-eval-guile-jobs: Adjust to multiple-output format.
[3] UL6TSQEA hydra-eval-guile-jobs: Use `--fresh-auto-compile'.
[4] 3K3GDTUQ hydra-eval-guile-jobs: Honor $NIX_STORE_DIR.
[5] YHDX3LB6 hydra-eval-guile-jobs: Add "This file is part of Hydra".
[6] 3HVZCXJR Shut up warnings
[7] UXRNODRJ Remove hydra-build and the old hydra-queue-runner
[8] ICYJNJWH Revert "Move build logic for the core hydra package into its own default.nix."
[9] STZE4KKR Fix build against Nix master
[10] HP5WJLQU hydra-eval-{jobs,jobset}: Pass file name as <...>
[11] AMFMXR52 Provide a command ‘hydra-init’ to initialise/upgrade the database
[12] Q6SOGMDQ Hydra/28: Rename "scheduler" to "evaluator"
[13] HXS5NUHW hydra-eval-guile-jobs: Adjust to <license> objects.
[14] 4CELXP7P Remove the longDescription field
[15] RXVJFQ5A Evaluator cleanups
[16] CQTN62OH Die tabs die
[17] ARD6Z67T Do incremental SVN checkouts
[18] DODOGD7M Send queue runner stats to statsd
[19] FV2M6MOT hydra: use autoconf/-make
[20] 4YCF3KBG Concurrent hydra-evaluator
[21] L4AI5YL6 Rename hydra_*.pl to hydra-*
[22] POPU2ATH * hydra_scheduler: use eval-jobs.
[23] VPAUFJTI Use PostgreSQL 9.5
[24] N22GPKYT * Put info about logs / build products in the DB.
[25] OOQ2D3KC * Refactoring: move fetchInput out of hydra_scheduler into a separate
[26] FJILMA6G configure: Preserve the user's $CPPFLAGS and $LDFLAGS.
[27] WQXF2T3D hydra-evaluator: Don't require $HYDRA_CONFIG
[28] XHOZT4WT Add a command `hydra-create-user' for managing user accounts
[29] QMW24O5S Add support for Guile & Guix.
[30] UGA45FNC Add a plugin for backing up builds in s3
[31] BXHG3HYL When renaming a jobset, add a redirect from the old name
[32] 6MGFQDR2 deleted some old scripts
[33] OVR2RWBI hydra-evaluator: Always pick the jobset that hasn't been evaluated longest
[34] FDE3BJAP * Refactoring.
[35] 3XTHEUMP * Implemented the clone feature.
[36] IE2PRAQU hydra-queue-runner: Send build notifications
[37] MPFSVI5X Pedantry: CLOCK_REALTIME -> CLOCK_MONOTONIC
[38] VU2OLHD2 Move most of AddBuilds to hydra-eval-jobset
[39] 2KJFOYGS hydra: Actually check for Nix in `configure'.
[40] KX5L74EY add nix-prefetch- scripts for now, were externals in svn
[41] FAIJDQKZ
[42] IS7GUIWY hydra-eval-guile-jobs: Register derivations as GC roots.
[43] IMQRX4MP hydra-eval-jobs: Use JSON instead of XML
[44] GBCPEGYC Add a .gitignore
[45] 5PXVIXLH Declarative projects: Include the fetch info for the declarative input in the project eval
[46] ODNCGFQ5 * Improved the navigation bar: don't include all projects (since that
[47] M3A5PZIH hydra: Clarify the dependency on BDW-GC.
[48] KNLKTCDM Use pkgconfig to find Nix
[49] QAJK5MCE Remove obsolete hydra-control script
[50] D5QIOJGP * Move everything up one directory.
[51] G2ZB6464 first test, not yet in buildprocess
[52] 3ZURHKX5 Update .gitignore
[53] 5SHCWE7X * Prevent repeated evaluation of a jobset with the same inputs. This
[54] LZVO64YG Merge in the first bits of the API work
[55] DBPIYHMA hydra: add nix-prefetch-* to tarball
[56] RFE6T5LG * Store jobset evaluations in the database explicitly. This includes
[57] NNQ7IQJX set scripts to executable
[58] E6QWSAFL Add hydra-dev-server which uses the classic Catalyst server
[59] TQVKZQUG Use the REST API in the web interface for editing jobsets
[60] UEMB7PTH Add option eval_substituter
[61] NS7SND6R hydra-evaluator: Send statistics to statsd
[62] SRBU2RAE Warn against multiple jobs with the same name
[63] Y6H7Y3OT Capture the path to `guile', when available.
[64] VML3XVFY hydra-eval-guile-jobs: Allow use of the installed Guix.
[65] HQGXL4MX Add validation for project and jobset names
[66] 6WRGCITD Enable declarative projects.
[67] HHWCS3QQ Jobs: populate Jobs.jobset_id field when writing from hydra-eval-jobset
[68] M2CFFNJY Remove unused file
[69] W2VB36NM guile: Raise default absolute timeout to from 2h to 20h.
[70] 5M6YEYUJ hydra-backfill-ids: create to add jobset_id values to Builds and Jobs
[71] 3ZCEPLNO
[*] T4LLYESZ * Nix expression for building Hydra.
[*] 3HZY24CX * Make jobsets viewable under

Change contents

edit in .gitignore at line 18

[11.178]→[11.0:34](∅→∅)

/src/script/hydra-eval-guile-jobs
edit in configure.ac at line 55

[11.56]→[11.60:61](∅→∅),[11.60]→[11.60:61](∅→∅),[11.39]→[11.0:75](∅→∅)

PKG_CHECK_MODULES([GUILE], [guile-2.0], [HAVE_GUILE=yes], [HAVE_GUILE=no])
edit in configure.ac at line 56

[11.1]→[11.35:147](∅→∅)

if test "x$HAVE_GUILE" = xyes; then
AC_PATH_PROG([GUILE], [guile])
else
GUILE="guile"
fi
AC_SUBST([GUILE])
edit in configure.ac at line 74

[11.16583]→[11.148:183](∅→∅)

src/script/hydra-eval-guile-jobs
replacement in configure.ac at line 78

[11.712]→[11.184:272](∅→∅)

AC_CONFIG_COMMANDS([executable-scripts],
[chmod +x src/script/hydra-eval-guile-jobs])

[11.712]

[6.34]

AC_CONFIG_COMMANDS([executable-scripts], [])
edit in release.nix at line 132

[8.2523]→[8.2523:2576](∅→∅)

guile # optional, for Guile + Guix support
edit in src/lib/Hydra/Controller/Jobset.pm at line 226

[11.1668]→[11.360:475](∅→∅),[11.475]→[11.11622:11697](∅→∅),[11.2367]→[11.543:544](∅→∅),[11.11697]→[11.543:544](∅→∅),[11.543]→[11.543:544](∅→∅)

# When the expression is in a .scm file, assume it's a Guile + Guix
# build expression.
my $exprType =
$c->stash->{params}->{"nixexprpath"} =~ /.scm$/ ? "guile" : "nix";
file deletion: hydra-eval-guile-jobs.in (----------)hydra-eval-guile-jobs (---r------)

[11.2543]→[11.667:715](∅→∅),[11.715]→[11.2679:2679](∅→∅),[11.2543]→[11.11339:11384](∅→∅),[11.11384]→[11.2679:2679](∅→∅)

#!/bin/sh
# Aside from this initial boilerplate, this is actually -*- scheme -*- code.
main="(module-ref (resolve-interface '(hydra-eval-guile-jobs)) 'eval-guile-jobs)"

# Keep the host's GUILE_LOAD_PATH unchanged to allow the installed Guix to
# be used. This moves Guix modules possibly out of control, but solves
# bootstrapping issues.
#
# Use `--fresh-auto-compile' to ignore any available .go, and force
# recompilation. This is because checkouts in the store has mtime set to
# the epoch, and thus .go files look newer, even though they may not
# correspond.

exec ${GUILE:-@GUILE@} --no-auto-compile --fresh-auto-compile \
-l "$0" -c "(apply $main (cdr (command-line)))" "$@"
!#
;;; Copyright © 2012, 2013, 2014 Ludovic Courtès <ludo@gnu.org>
;;;
;;; This file is part of Hydra.
;;;
;;; Hydra is free software: you can redistribute it and/or modify
;;; it under the terms of the GNU General Public License as published by
;;; the Free Software Foundation, either version 3 of the License, or
;;; (at your option) any later version.
;;;
;;; Hydra is distributed in the hope that it will be useful,
;;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with Hydra. If not, see <http://www.gnu.org/licenses/>.

(define-module (hydra-eval-guile-jobs)
#:use-module (sxml simple)
#:use-module (ice-9 match)
#:use-module (ice-9 regex)
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-11)
#:export (job-evaluations->xml
eval-guile-jobs))

(define (guix-variable module name)
"Dynamically link variable NAME under Guix module MODULE and return it.
Note: this is used instead of `@', because when using `@' in an uncompiled
file, Guile tries to load the module directly as it reads the source, which
fails in our case, leading to the creation of empty (guix ...) modules."
;; TODO: fail with an XML error description
(let ((m (resolve-interface `(guix ,module))))
(module-ref m name)))

(define (%derivation-system drv)
;; XXX: Awful hack to workaround the fact that `derivation-system', which
;; is a macro, cannot be referred to dynamically.
(struct-ref drv 3))

(define strip-store-path
(let* ((store (or (getenv "NIX_STORE_DIR") "/nix/store"))
(store-path-rx
(make-regexp (string-append "^.*" (regexp-quote store)
"/[^-]+-(.+)$"))))
(lambda (path)
(or (and=> (regexp-exec store-path-rx path)
(lambda (match)
(let ((path (match:substring match 1)))
path)))
path))))

(define (derivation-path->name drv)
"Return the base name of DRV, sans hash and `.drv' extension."
(let ((d (strip-store-path drv)))
(if (string-suffix? ".drv" d)
(string-drop-right d 4)
d)))

(define (register-gc-root drv roots-dir)
"Register a permanent garbage collector root under ROOTS-DIR for DRV."
(let ((root (string-append roots-dir "/" (basename drv))))
(unless (file-exists? root)
(symlink drv root))))

(define* (job-evaluations->sxml jobs
#:key gc-roots-dir)
"Return the hydra-eval-jobs SXML form for the result of JOBS, a list of
symbol/thunk pairs."
`(*TOP*
(*PI* xml "version='1.0' encoding='utf-8'")
"\n"
(jobs "\n"
,@(map (match-lambda
(((? symbol? name) . (? thunk? thunk))
(let* ((result (save-module-excursion
(lambda ()
(set-current-module %user-module)
(with-output-to-port (%make-void-port "w")
thunk))))
(drv (assoc-ref result 'derivation)))
(define (opt-attr xml-name name)
(match (assoc name result)
((_ . value)
`((,xml-name ,value)))
(_
'())))

;; XXX: Add <arg ...> tags?
`(job (@ (jobName ,name)
(drvPath ,drv)
,@(opt-attr 'homepage 'home-page)
(license
,(let loop ((license (assoc-ref result 'license)))
(match license
((? struct?)
(struct-ref license 0))
((l ...)
(string-join (map loop l)))
(_ ""))))
,@(opt-attr 'description 'description)
(maintainers
,(string-join (or (assoc-ref result 'maintainers)
'())
", "))
(maxSilent
,(number->string (or (assoc-ref result
'max-silent-time)
3600)))
(timeout
,(number->string (or (assoc-ref result 'timeout)
72000)))
(nixName ,(derivation-path->name drv))
(schedulingPriority
,(number->string (or (assoc-ref result
'scheduling-priority)
10)))
(system
,(call-with-input-file drv
(compose %derivation-system
(guix-variable 'derivations
'read-derivation)))))
"\n"))))
jobs))))

(define* (job-evaluations->xml jobs port
#:key gc-roots-dir)
(set-port-encoding! port "UTF-8")
(sxml->xml (job-evaluations->sxml jobs #:gc-roots-dir gc-roots-dir)
port))

;;;
;;; Command-line entry point.
;;;

(define (parse-arguments args)
"Traverse ARGS, a list of command-line arguments compatible with
`hydra-eval-jobs', and return the name of the file that defines the jobs, an
expression that returns the entry point in that file (a unary procedure), the
list of name/value pairs passed to that entry point, as well as a GC root
directory or #f."
(define (module-directory dir)
(let ((d (string-append dir "/share/guile/site/2.0")))
(if (file-exists? d)
d
dir)))

(let loop ((args args)
(result '())
(file #f)
(entry 'hydra-jobs)
(roots-dir #f))
(match args
(()
(if (not file)
(error "hydra-eval-guile-jobs: no expression file given")
(values file entry (reverse result) roots-dir)))
(("-I" name=dir rest ...)
(let* ((dir (match (string-tokenize name=dir
(char-set-complement (char-set
#\=)))
((_ dir) dir)
((dir) dir)))
(dir* (module-directory dir)))
(format (current-error-port) "adding `~a' to the load path~%" dir*)
(set! %load-path (cons dir* %load-path))
(set! %load-compiled-path (cons dir* %load-compiled-path)))
(loop rest result file entry roots-dir))
(("--argstr" name value rest ...)
(loop rest (alist-cons (string->symbol name) value result)
file entry roots-dir))
(("--arg" name expr rest ...)
(let ((value (eval (call-with-input-string expr read)
(current-module))))
(loop rest (alist-cons (string->symbol name) value result)
file entry roots-dir)))
(("--gc-roots-dir" dir rest ...)
(loop rest result file entry dir))
(("-j" _ rest ...) ; XXX: what's this?
(loop rest result file entry roots-dir))
(("--entry" expr rest ...) ; entry point, like `guile -e'
(let ((expr (call-with-input-string expr read)))
(loop rest result file expr roots-dir)))
((file rest ...) ; source file that defines the jobs
(loop rest result file entry roots-dir))
(_
(error "hydra-eval-guile-jobs: invalid arguments" args)))))

(define %user-module
;; Hydra user module.
;; TODO: Make it a sandbox.
(let ((m (make-module)))
(beautify-user-module! m)
m))

(define (eval-guile-jobs . args)
(setlocale LC_ALL "")

(let-values (((file entry args gc-roots-dir)
(parse-arguments args)))

(save-module-excursion
(lambda ()
(set-current-module %user-module)

;; The standard output must contain only XML.
(with-output-to-port (%make-void-port "w")
(lambda ()
(primitive-load file)))))

(let* ((entry (eval entry %user-module))
(store ((guix-variable 'store 'open-connection)))
(jobs (entry store args)))
(unless (string? gc-roots-dir)
(format (current-error-port)
"warning: --gc-roots-dir not specified~%"))

(job-evaluations->xml jobs (current-output-port)
#:gc-roots-dir gc-roots-dir))))
;; Resolve Guix modules lazily.
,(map (match-lambda
((name . path)
`(output (@ (name ,name) (path ,path)))))
((guix-variable 'derivations
'derivation-path->output-paths)
drv))

(when gc-roots-dir
;; Register DRV as a GC root so that it's not collected by
;; the time 'hydra-queue-runner' attempts to build it.
(register-gc-root drv gc-roots-dir))
replacement in src/script/Makefile.am at line 2

[11.293]→[11.293:324](∅→∅),[11.346]→[11.346:373](∅→∅)

$(distributable_scripts) \
hydra-eval-guile-jobs.in

[11.293]

[11.97]

$(distributable_scripts)
replacement in src/script/Makefile.am at line 19

[11.611]→[11.611:666](∅→∅)

$(distributable_scripts) \
hydra-eval-guile-jobs

[11.611]

$(distributable_scripts)
replacement in src/script/hydra-eval-jobset at line 267

[11.7572]→[11.7572:7859](∅→∅)

my ($exprType, $value) = @_;
my $result;
if ($exprType eq "guile") {
if ($value eq "true") {
$result = "#t";
} else {
$result = "#f";
}
$result = $value;
} else {
$result = $value;
}
return $result;

[11.7572]

[11.7859]

my ($value) = @_;
return $value;
replacement in src/script/hydra-eval-jobset at line 273

[11.7888]→[11.7888:9575](∅→∅)

my ($exprType, $input) = @_;
my $result;
if ($exprType eq "guile") {
$result = "'((file-name . \"" . ${input}->{storePath} . "\")" .
(defined $input->{revision} ? "(revision . \"" . $input->{revision} . "\")" : "") .
(defined $input->{revCount} ? "(revision-count . " . $input->{revCount} . ")" : "") .
(defined $input->{gitTag} ? "(git-tag . \"" . $input->{gitTag} . "\")" : "") .
(defined $input->{shortRev} ? "(short-revision . \"" . $input->{shortRev} . "\")" : "") .
(defined $input->{version} ? "(version . \"" . $input->{version} . "\")" : "") .
")";
} else {
$result = "{ outPath = builtins.storePath " . $input->{storePath} . "" .
"; inputType = \"" . $input->{type} . "\"" .
(defined $input->{uri} ? "; uri = \"" . $input->{uri} . "\"" : "") .
(defined $input->{revNumber} ? "; rev = " . $input->{revNumber} . "" : "") .
(defined $input->{revision} ? "; rev = \"" . $input->{revision} . "\"" : "") .
(defined $input->{revCount} ? "; revCount = " . $input->{revCount} . "" : "") .
(defined $input->{gitTag} ? "; gitTag = \"" . $input->{gitTag} . "\"" : "") .
(defined $input->{shortRev} ? "; shortRev = \"" . $input->{shortRev} . "\"" : "") .
(defined $input->{version} ? "; version = \"" . $input->{version} . "\"" : "") .
(defined $input->{outputName} ? "; outputName = \"" . $input->{outputName} . "\"" : "") .
(defined $input->{drvPath} ? "; drvPath = builtins.storePath " . $input->{drvPath} . "" : "") .
";}";
}
return $result;

[11.7888]

[11.9575]

my ($input) = @_;
return
"{ outPath = builtins.storePath " . $input->{storePath} . "" .
"; inputType = \"" . $input->{type} . "\"" .
(defined $input->{uri} ? "; uri = \"" . $input->{uri} . "\"" : "") .
(defined $input->{revNumber} ? "; rev = " . $input->{revNumber} . "" : "") .
(defined $input->{revision} ? "; rev = \"" . $input->{revision} . "\"" : "") .
(defined $input->{revCount} ? "; revCount = " . $input->{revCount} . "" : "") .
(defined $input->{gitTag} ? "; gitTag = \"" . $input->{gitTag} . "\"" : "") .
(defined $input->{shortRev} ? "; shortRev = \"" . $input->{shortRev} . "\"" : "") .
(defined $input->{version} ? "; version = \"" . $input->{version} . "\"" : "") .
(defined $input->{outputName} ? "; outputName = \"" . $input->{outputName} . "\"" : "") .
(defined $input->{drvPath} ? "; drvPath = builtins.storePath " . $input->{drvPath} . "" : "") .
";}";
replacement in src/script/hydra-eval-jobset at line 291

[11.9598]→[11.9598:9635](∅→∅)

my ($inputInfo, $exprType) = @_;

[11.9598]

[11.9635]

my ($inputInfo) = @_;
replacement in src/script/hydra-eval-jobset at line 308

[9.1227]→[9.1227:1310](∅→∅)

push @res, "--arg", $input, booleanToString($exprType, $alt->{value});

[9.1227]

[9.1310]

push @res, "--arg", $input, booleanToString($alt->{value});
edit in src/script/hydra-eval-jobset at line 311

[9.1360]→[9.1360:1465](∅→∅)

die "input type ‘nix’ only supported for Nix-based jobsets\n" unless $exprType eq "nix";
edit in src/script/hydra-eval-jobset at line 314

[9.1571]→[9.1571:1677](∅→∅)

die "input type ‘eval’ only supported for Nix-based jobsets\n" unless $exprType eq "nix";
replacement in src/script/hydra-eval-jobset at line 322

[9.1975]→[9.1975:2052](∅→∅)

push @res, "--arg", $input, buildInputToString($exprType, $alt);

[9.1975]

[9.2052]

push @res, "--arg", $input, buildInputToString($alt);
replacement in src/script/hydra-eval-jobset at line 331

[11.11074]→[11.11074:11144](∅→∅)

my ($inputInfo, $exprType, $nixExprInputName, $nixExprPath) = @_;

[11.11074]

[11.1424]

my ($inputInfo, $nixExprInputName, $nixExprPath) = @_;
replacement in src/script/hydra-eval-jobset at line 336

[11.11509]→[11.11509:11600](∅→∅),[11.11600]→[10.100:127](∅→∅)

my $evaluator = ($exprType eq "guile") ? "hydra-eval-guile-jobs" : "hydra-eval-jobs";

my @cmd = ($evaluator,

[11.11509]

[10.127]

my @cmd = ("hydra-eval-jobs",
replacement in src/script/hydra-eval-jobset at line 340

[10.266]→[10.266:319](∅→∅)

inputsToArgs($inputInfo, $exprType));

[10.266]

[11.11725]

inputsToArgs($inputInfo));
replacement in src/script/hydra-eval-jobset at line 353

[11.12055]→[11.12055:12146](∅→∅)

die "$evaluator returned " . ($res & 127 ? "signal $res" : "exit code " . ($res >> 8))

[11.12055]

[11.12146]

die "hydra-eval-jobs returned " . ($res & 127 ? "signal $res" : "exit code " . ($res >> 8))
edit in src/script/hydra-eval-jobset at line 572

[11.3692]→[11.11385:11455](∅→∅)

my $exprType = $jobset->nixexprpath =~ /.scm$/ ? "guile" : "nix";
replacement in src/script/hydra-eval-jobset at line 597

[11.2206]→[11.11456:11555](∅→∅)

my @args = ($jobset->nixexprinput, $jobset->nixexprpath, inputsToArgs($inputInfo, $exprType));

[11.2206]

[11.2294]

my @args = ($jobset->nixexprinput, $jobset->nixexprpath, inputsToArgs($inputInfo));
replacement in src/script/hydra-eval-jobset at line 612

[11.217]→[11.2570:2680](∅→∅),[11.534]→[11.2570:2680](∅→∅),[11.5549]→[11.2570:2680](∅→∅)

my ($jobs, $nixExprInput) = evalJobs($inputInfo, $exprType, $jobset->nixexprinput, $jobset->nixexprpath);

[11.217]

[11.218]

my ($jobs, $nixExprInput) = evalJobs($inputInfo, $jobset->nixexprinput, $jobset->nixexprpath);