grahamc/hydra2 - Change ZZYTUBG2HVWNYS5P3FHKK5L5CYN4ZVXPSUE36A27KPHOL6UPJHEAC

Fix extreme slowness in hydra-queue-runner

If there are builds in the queue that depend on another scheduled build, then hydra-queue-runner will start the dependency first and block the dependent builds. This is implemented in findBuildDependencyInQueue. However, if there are tens of thousands of such dependent builds, since each call to findBuildDependencyInQueue may take a second or so, hydra-queue-runner will spend hours just deciding which builds not to do. Thus very little progress is made.

So now, when a build is started, we immediately check which builds are "blocked" by it (i.e. depend on it), and remove such builds from consideration.

Created by Eelco Dolstra on October 11, 2013

ZZYTUBG2HVWNYS5P3FHKK5L5CYN4ZVXPSUE36A27KPHOL6UPJHEAC

Dependencies

In channels

main

Change contents

Insertion in src/script/hydra-queue-runner at line 12 [4.7823]
[3.420]
[4.7878]
```
use Set::Scalar;
```

Replacement in src/script/hydra-queue-runner at line 56 [4.7823]

B:BD[5.65] → [5.65:184]

        my $b = $buildsByDrv->{$d};
        next unless defined $b;
        return $db->resultset('Builds')->find($b);

[5.65]

[5.184]

        my $bs = $buildsByDrv->{$d};
        next unless defined $bs;
        return $db->resultset('Builds')->find((@$bs)[0]);

Insertion in src/script/hydra-queue-runner at line 64 [4.7823]

[6.369]

[7.451]

sub blockBuilds {
    my ($buildsByDrv, $blockedBuilds, $build) = @_;
    my @rdeps = grep { /\.drv$/ && $_ ne $build->drvpath } computeFSClosure(1, 0, $build->drvpath);
    foreach my $drv (@rdeps) {
        my $bs = $buildsByDrv->{$drv};
        next if !defined $bs;
        $blockedBuilds->insert($_) foreach @$bs;
    }
}

Replacement in src/script/hydra-queue-runner at line 95 [4.7823]

B:BD[5.334] → [5.334:379]

B:BD[5.379] → [8.3198:3296]

        $buildsByDrv->{$_->drvpath} = $_->id
            foreach $db->resultset('Builds')->search({ finished => 0 }, { join => ['project'] });

[5.334]

[5.491]

        push @{$buildsByDrv->{$_->drvpath}}, $_->id
            foreach $db->resultset('Builds')->search({ finished => 0 });
        # Builds in the queue of which a dependency is already building.
        my $blockedBuilds = Set::Scalar->new();
        blockBuilds($buildsByDrv, $blockedBuilds, $_)
            foreach $db->resultset('Builds')->search({ finished => 0, busy => 1 });

Insertion in src/script/hydra-queue-runner at line 182 [4.7823]

[8.5931]

                        next if $blockedBuilds->has($build->id);

Insertion in src/script/hydra-queue-runner at line 209 [4.7823]

[2.1259]

[8.7220]

                        blockBuilds($buildsByDrv, $blockedBuilds, $build);