grahamc/hydra - Change RXVJFQ5AV3WME4HDVBPSRCALQTXROT4KQPOQVO6KTWTBNZIZZGPAC

Evaluator cleanups

Don't use isCurrent anymore; instead look up builds in the previous jobset evaluation. (The isCurrent field is still maintained because it's still used in some other places.)
To determine whether to perform an evaluation, compare the hash of the current inputs with the inputs of the previous jobset evaluation, rather than checking if there was ever an evaluation with those inputs. This way, if the inputs of an evaluation change back to a previous state, we get a new jobset evaluation in the database (and thus the latest jobset evaluation correctly represents the latest state of the jobset).
Improve performance by removing some unnecessary operations and adding an index.

Created by Eelco Dolstra on March 7, 2012

RXVJFQ5AV3WME4HDVBPSRCALQTXROT4KQPOQVO6KTWTBNZIZZGPAC

Dependencies

In channels

upstream

main

Change contents

Replacement in src/lib/Hydra/Controller/Build.pm at line 541 [9.1]

B:BD[8.1031] → [8.1031:1090]

        $inputInfo, $nixExprInput, $job, \%currentBuilds);

[8.1031]

[8.1090]

        $inputInfo, $nixExprInput, $job, \%currentBuilds, undef);

Insertion in src/lib/Hydra/Helper/AddBuilds.pm at line 20 [11.2663]
[10.156]
[10.156]
```
    getPrevJobsetEval
```

Insertion in src/lib/Hydra/Helper/AddBuilds.pm at line 788 [11.2663]

[10.1513]

[12.2316]

}
# Return the most recent evaluation of the given jobset that had new
# builds, or undefined if no such evaluation exists.
sub getPrevJobsetEval {
    my ($db, $jobset) = @_;
    my ($prevEval) = $jobset->jobsetevals({ hasnewbuilds => 1 }, { order_by => "id DESC", rows => 1 });
    return $prevEval;

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 802 [11.2663]

B:BD[8.3507] → [8.3507:3600]

    my ($db, $project, $jobset, $inputInfo, $nixExprInput, $buildInfo, $currentBuilds) = @_;

[8.3507]

[8.3600]

    my ($db, $project, $jobset, $inputInfo, $nixExprInput, $buildInfo, $buildIds, $prevEval) = @_;

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 835 [11.2663]

B:BD[8.4892] → [8.4892:5079]

B:BD[8.5079] → [13.641:707]

∅:D[13.707] → [8.5145:5165]

B:BD[8.5145] → [8.5145:5165]

        my @previousBuilds = $job->builds->search({outPath => $outPath, isCurrent => 1});
        if (scalar(@previousBuilds) > 0) {
            print STDERR "already scheduled/built\n";
            $currentBuilds->{$_->id} = 0 foreach @previousBuilds;
            return;

[8.4892]

[8.5165]

        if (defined $prevEval) {
            my ($prevBuild) = $prevEval->builds->search({ job => $job->name, outPath => $outPath }, { rows => 1, columns => ['id'] });
            if (defined $prevBuild) {
                print STDERR "    already scheduled/built as build ", $prevBuild->id, "\n";
                $buildIds->{$prevBuild->id} = 0;
                return;
            }

Insertion in src/lib/Hydra/Helper/AddBuilds.pm at line 847 [11.2663]

[8.5211]

        my %extraFlags;
        if (isValidPath($outPath)) {
            %extraFlags =
                ( finished => 1 
                , iscachedbuild => 1
                , buildstatus => 0
                , starttime => $time 
                , stoptime => $time 
                , logfile => getBuildLog($drvPath)
                , errormsg => ""
                , releasename => getReleaseName($outPath)
                );
        } else {
            %extraFlags = ( finished => 0 );
        }

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 864 [11.2663]
B:BD[8.5250] → [8.5250:5278]
B:BD[8.5278] → [14.439:473]
```
            { finished => 0
            , timestamp => $time 
```
[8.5250]
[8.5312]
```
            { timestamp => $time 
```

Insertion in src/lib/Hydra/Helper/AddBuilds.pm at line 879 [11.2663]

[8.5872]

	    , priority => $priority
	    , busy => 0
	    , locker => ""
	    , %extraFlags

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 885 [11.2663]
B:BD[8.5965] → [8.5965:6007]
```
        $currentBuilds->{$build->id} = 1;
```
[8.5889]
[8.6007]
```
        $buildIds->{$build->id} = 1;
```

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 887 [11.2663]

B:BD[8.6016] → [5.1426:1463]

∅:D[5.1463] → [14.510:580]

B:BD[14.510] → [14.510:580]

B:BD[14.580] → [15.4619:4710]

∅:D[15.4710] → [14.715:987]

B:BD[14.715] → [14.715:987]

        if (isValidPath($outPath)) {
            print STDERR "marked as cached build ", $build->id, "\n";
	    $build->update(
                { finished => 1 
                , iscachedbuild => 1
                , buildstatus => 0
                , starttime => $time 
                , stoptime => $time 
                , logfile => getBuildLog($drvPath)
                , errormsg => ""
                , releasename => getReleaseName($outPath)
                });

[8.6016]

[3.0]

        if ($build->iscachedbuild) {
            print STDERR "    marked as cached build ", $build->id, "\n";

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 891 [11.2663]

B:BD[14.1004] → [14.1004:1075]

B:BD[14.1075] → [16.2942:2989]

∅:D[16.2989] → [14.1173:1252]

B:BD[14.1173] → [14.1173:1252]

            print STDERR "added to queue as build ", $build->id, "\n";
	    $build->update(
		{ priority => $priority
                , busy => 0
                , locker => ""
                });

[14.1004]

[14.1252]

            print STDERR "    added to queue as build ", $build->id, "\n";

Deletion in src/lib/Hydra/Schema/JobsetEvals.pm at line 164 [17.65]
B:BD[17.1278] → [13.4789:4879]
```
# You can replace this text with custom content, and it will be preserved on regeneration
```

Replacement in src/script/hydra-evaluator at line 83 [19.1319]

B:BD[18.1440] → [18.1440:1496]

    print $email->as_string if $ENV{'HYDRA_MAIL_TEST'};

[18.1440]

[18.1496]

    print STDERR $email->as_string if $ENV{'HYDRA_MAIL_TEST'};

Insertion in src/script/hydra-evaluator at line 87 [19.1319]
[18.1521]
[18.1521]

Replacement in src/script/hydra-evaluator at line 109 [19.1319]

∅:D[4.295] → [17.2104:2174]

B:BD[17.2104] → [17.2104:2174]

    # JobsetInputHashes to see if we've already evaluated this set of

[4.295]

[17.2174]

    # JobsetInputHashes to see if the previous evaluation had the same

Replacement in src/script/hydra-evaluator at line 113 [19.1319]

B:BD[17.2334] → [17.2334:2335]

B:BD[17.2335] → [6.412:485]

∅:D[6.485] → [17.2400:2449]

∅:D[13.5522] → [17.2400:2449]

B:BD[17.2400] → [17.2400:2449]


    if (scalar($jobset->jobsetevals->search({hash => $argsHash})) > 0) {
        print "  already evaluated, skipping\n";

[17.2334]

[17.2449]

    my $prevEval = getPrevJobsetEval($db, $jobset);
    if ($prevEval->hash eq $argsHash) {
        print STDERR "  jobset is unchanged, skipping\n";

Insertion in src/script/hydra-evaluator at line 129 [19.1319]

[20.5278]

[13.5576]

        # Clear the "current" flag on all builds.  Since we're in a
        # transaction this will only become visible after the new
        # current builds have been added.
        $jobset->builds->search({iscurrent => 1})->update({iscurrent => 0});

Replacement in src/script/hydra-evaluator at line 135 [19.1319]
B:BD[13.5628] → [13.5628:5655]
```
        my %currentBuilds;
```
[13.5628]
[13.5655]
```
        my %buildIds;
```

Replacement in src/script/hydra-evaluator at line 138 [19.1319]

B:BD[13.5750] → [13.5750:5911]

            print "considering job " . $job->{jobName} . "\n";
            checkBuild($db, $project, $jobset, $inputInfo, $nixExprInput, $job, \%currentBuilds);

[13.5750]

[13.5911]

            print STDERR "  considering job " . $project->name, ":", $jobset->name, ":", $job->{jobName} . "\n";
            checkBuild($db, $project, $jobset, $inputInfo, $nixExprInput, $job, \%buildIds, $prevEval);

Replacement in src/script/hydra-evaluator at line 149 [19.1319]

B:BD[21.2042] → [22.1183:1364]

∅:D[22.1364] → [23.6513:6534]

B:BD[23.6513] → [23.6513:6534]

B:BD[23.6534] → [22.1365:1416]

∅:D[22.1416] → [23.6589:6613]

B:BD[23.6589] → [23.6589:6613]

        foreach my $job ($jobset->jobs->all) {
            if ($failedJobNames{$job->name}) {
                $job->update({errormsg => join '\n', @{$failedJobNames{$job->name}}});
            } else {
                $job->update({errormsg => undef});
            }
        }

[21.2042]

[20.5548]

        $_->update({ errormsg => $failedJobNames{$_->name} ? join '\n', @{$failedJobNames{$_->name}} : undef })
            foreach $jobset->jobs->all;

Deletion in src/script/hydra-evaluator at line 152 [19.1319]

B:BD[20.5549] → [20.5549:5708]

B:BD[20.5708] → [2.0:88]

∅:D[2.88] → [13.5923:5934]

B:BD[20.5839] → [13.5923:5934]

        # Clear the "current" flag on all builds that are no longer
        # current.
        foreach my $build ($jobset->builds->search({iscurrent => 1})) {
            $build->update({iscurrent => 0}) unless defined $currentBuilds{$build->id};
        }

Replacement in src/script/hydra-evaluator at line 153 [19.1319]

B:BD[13.5964] → [13.5964:6019]

        while (my ($id, $new) = each %currentBuilds) {

[13.5964]

[13.6019]

        while (my ($id, $new) = each %buildIds) {

Replacement in src/script/hydra-evaluator at line 166 [19.1319]

B:BD[13.6380] → [13.6380:6439]

            while (my ($id, $new) = each %currentBuilds) {

[13.6380]

[13.6439]

            while (my ($id, $new) = each %buildIds) {

Replacement in src/script/hydra-evaluator at line 172 [19.1319]
∅:D[20.5866] → [24.689:755]
∅:D[23.6630] → [24.689:755]
B:BD[24.689] → [24.689:755]
```
    # Store the errors messages for jobs that failed to evaluate.
```
[20.5866]
[24.755]
```
    # Store the error messages for jobs that failed to evaluate.
```

Replacement in src/script/hydra-evaluator at line 195 [19.1319]

B:BD[21.2106] → [21.2106:2184]

    print "considering jobset ", $jobset->name, " in ", $project->name, "\n";

[21.2106]

[21.2184]

    print STDERR "considering jobset ", $project->name, ":", $jobset->name, "\n";

Replacement in src/script/hydra-evaluator at line 203 [19.1319]

B:BD[21.2288] → [21.2288:2355]

        print "error evaluating jobset ", $jobset->name, ": $msg";

[21.2288]

[25.1624]

        print STDERR "error evaluating jobset ", $jobset->name, ": $msg";

Replacement in src/script/hydra-evaluator at line 214 [19.1319]

B:BD[26.1831] → [27.4407:4467]

        print "considering project ", $project->name, "\n";

[26.1831]

[28.3184]

        print STDERR "considering project ", $project->name, "\n";

Replacement in src/script/hydra-evaluator at line 235 [19.1319]
B:BD[29.37] → [29.37:65]
∅:D[29.65] → [30.995:1022]
B:BD[30.995] → [30.995:1022]
```
    if ($@) { print "$@"; }
    print "sleeping...\n";
```
[29.37]
[31.855]
```
    if ($@) { print STDERR "$@"; }
    print STDERR "sleeping...\n";
```
Insertion in src/sql/hydra.sql at line 496 [19.1]
[32.2228]
[33.2113]

Deletion in src/sql/hydra.sql at line 521 [19.1]

B:BD[34.54] → [7.3042:3274]

create index IndexBuildsOnKeep on Builds(keep); -- used by hydra-update-gc-roots
create index IndexMostRecentSuccessfulBuilds on Builds(project, jobset, job, system, finished, buildStatus, id desc); -- used by hydra-update-gc-roots

Insertion in src/sql/hydra.sql at line 530 [19.1]

[35.139]


--  For hydra-update-gc-roots.
create index IndexBuildsOnKeep on Builds(keep);
create index IndexMostRecentSuccessfulBuilds on Builds(project, jobset, job, system, finished, buildStatus, id desc);
-- To get the most recent eval for a jobset.
create index IndexJobsetEvalsOnJobsetId on JobsetEvals(project, jobset, hasNewBuilds, id desc);

File addition: upgrade-4.sql (----------)

[36.3004]

create index IndexJobsetEvalsOnJobsetId on JobsetEvals(project, jobset, hasNewBuilds, id desc);