declarative projects: support fully static, declarative configuration

[?]

Sep 2, 2020, 4:34 PM

BCFAS6VOEDLANURVQYMHK4FQY5JMBD6U2KMRZKS2T73LBP4T3QIAC

Dependencies

[2] 5PXVIXLH Declarative projects: Include the fetch info for the declarative input in the project eval
[3] VDT6CUIT doc: add basic docs about declarative projects
[4] CUFVKLLA Remove Hydra::Helper::nix::txn_do from the Perl code
[5] CQZQE32V Improve handling of Perl's block eval errors
[6] HEZQ273S improve the error messages when invalid declarative jobsets are defined
[7] 6WRGCITD Enable declarative projects.
[8] L54KA7IM handleDeclarativeJobsetBuild: handle errors from readNixFile
[*] OOQ2D3KC * Refactoring: move fetchInput out of hydra_scheduler into a separate
[*] BMSQD2ZH Indentation
[*] N22GPKYT * Put info about logs / build products in the DB.

Change contents

edit in doc/manual/declarative-projects.xml at line 4

[3.145]

[3.145]

<title>Declarative projects</title>

<para>
Hydra supports declaratively configuring a project's jobsets. This
configuration can be done statically, or generated by a build job.
</para>

<note><para>
Hydra will treat the project's declarative input as a static definition
if and only if the spec file contains a dictionary of dictionaries.
If the value of any key in the spec is not a dictionary, it will
treat the spec as a generated declarative spec.
</para></note>

<section xml:id="sec-static-declarative-projects">

<title>Static, Declarative Projects</title>
<para>
Hydra supports declarative projects, where jobsets are configured
from a static JSON document in a repository.
</para>

<para>
To configure a static declarative project, take the following steps:
</para>
<orderedlist numeration="arabic" spacing="compact">
<listitem>
<para>
Create a Hydra-fetchable source like a Git repository or local path.
</para>
</listitem>
<listitem>
<para>
In that source, create a file called <filename>spec.json</filename>,
and add the specification for all of the jobsets. Each key is jobset
and each value is a jobset's specification. For example:
replacement in doc/manual/declarative-projects.xml at line 42

[3.146]→[3.146:184](∅→∅)

<title>Declarative projects</title>

[3.146]

[3.184]

<programlisting language="json">
{
"nixpkgs": {
"enabled": 1,
"hidden": false,
"description": "Nixpkgs",
"nixexprinput": "nixpkgs",
"nixexprpath": "pkgs/top-level/release.nix",
"checkinterval": 300,
"schedulingshares": 100,
"enableemail": false,
"emailoverride": "",
"keepnr": 3,
"inputs": {
"nixpkgs": {
"type": "git",
"value": "git://github.com/NixOS/nixpkgs.git master",
"emailresponsible": false
}
}
},
"nixos": {
"enabled": 1,
"hidden": false,
"description": "NixOS: Small Evaluation",
"nixexprinput": "nixpkgs",
"nixexprpath": "nixos/release-small.nix",
"checkinterval": 300,
"schedulingshares": 100,
"enableemail": false,
"emailoverride": "",
"keepnr": 3,
"inputs": {
"nixpkgs": {
"type": "git",
"value": "git://github.com/NixOS/nixpkgs.git master",
"emailresponsible": false
}
}
}
}
</programlisting>
</para>
</listitem>
<listitem>
<para>
Create a new project, and set the project's declarative input type,
declarative input value, and declarative spec file to point to the
source and JSON file you created in step 2.
</para>
</listitem>
</orderedlist>
replacement in doc/manual/declarative-projects.xml at line 95

[3.193]→[3.193:342](∅→∅)

Hydra also supports declarative projects, where jobsets are generated
and configured automatically from specification files instead of being

[3.193]

[3.342]

Hydra will create a special jobset named <literal>.jobsets</literal>.
When the <literal>.jobsets</literal> jobset is evaluated, this static
specification will be used for configuring the rest of the project's
jobsets.
</para>
</section>

<section xml:id="sec-generated-declarative-projects">

<title>Generated, Declarative Projects</title>
<para>
<para>
Hydra also supports generated declarative projects, where jobsets are
configured automatically from specification files instead of being
edit in doc/manual/declarative-projects.xml at line 197

[3.3832]

</section>
edit in src/lib/Hydra/Helper/AddBuilds.pm at line 24

[6.802]

[11.156]

handleDeclarativeJobsetJson
edit in src/lib/Hydra/Helper/AddBuilds.pm at line 69

[6.1919]

[6.1919]

sub handleDeclarativeJobsetJson {
my ($db, $project, $declSpec) = @_;
$db->txn_do(sub {
my @kept = keys %$declSpec;
push @kept, ".jobsets";
$project->jobsets->search({ name => { "not in" => \@kept } })->update({ enabled => 0, hidden => 1 });
while ((my $jobsetName, my $spec) = each %$declSpec) {
eval {
updateDeclarativeJobset($db, $project, $jobsetName, $spec);
1;
} or do {
print STDERR "ERROR: failed to process declarative jobset ", $project->name, ":${jobsetName}, ", $@, "\n";
}
}
});
}
replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 102

[6.2344]→[4.692:718](∅→∅),[4.718]→[6.2370:2627](∅→∅),[6.2370]→[6.2370:2627](∅→∅),[6.2627]→[6.0:103](∅→∅),[6.103]→[5.106:155](∅→∅),[5.155]→[6.148:293](∅→∅),[6.148]→[6.148:293](∅→∅),[6.293]→[6.2703:2729](∅→∅),[6.2703]→[6.2703:2729](∅→∅)

$db->txn_do(sub {
my @kept = keys %$declSpec;
push @kept, ".jobsets";
$project->jobsets->search({ name => { "not in" => \@kept } })->update({ enabled => 0, hidden => 1 });
while ((my $jobsetName, my $spec) = each %$declSpec) {
eval {
updateDeclarativeJobset($db, $project, $jobsetName, $spec);
1;
} or do {
print STDERR "ERROR: failed to process declarative jobset ", $project->name, ":${jobsetName}, ", $@, "\n";
}
}
});

[6.2344]

[5.156]

handleDeclarativeJobsetJson($db, $project, $declSpec);
edit in src/script/hydra-eval-jobset at line 572

[6.5233]

[6.5233]
replacement in src/script/hydra-eval-jobset at line 574

[6.5316]→[6.5316:5421](∅→∅),[6.5421]→[2.25:77](∅→∅)

updateDeclarativeJobset($db, $project, ".jobsets", $declSpec);
$jobset->discard_changes;
$inputInfo->{"declInput"} = [ $declInput ];

[6.5316]

[6.5421]

if (ref $declSpec eq "HASH") {
if (grep ref $_ eq "HASH", values %$declSpec) {
# Since all of its keys are hashes, assume the json document
# itself is the entire set of jobs
handleDeclarativeJobsetJson($db, $project, $declSpec);
$db->txn_do(sub {
$jobset->update({ lastcheckedtime => time, fetcherrormsg => undef });
});
return;
} else {
# Update the jobset with the spec's inputs, and the continue
# evaluating the .jobsets jobset.
updateDeclarativeJobset($db, $project, ".jobsets", $declSpec);
$jobset->discard_changes;
$inputInfo->{"declInput"} = [ $declInput ];
}
} else {
die "Declarative specification file $declFile is not a dictionary"
}