grahamc/hydra - Change GEADFVZ5LXXFIE3VIP4UJ4AEI2VX57DXER47JA4IHH5BG3QNPAEAC

hydra-queue-runner: Improved scheduling

Each jobset now has a "scheduling share" that determines how much of the build farm's time it is entitled to. For instance, if a jobset has 100 shares and the total number of shares of all jobsets is 1000, it's entitled to 10% of the build farm's time. When there is a free build slot for a given system type, the queue runner will select the jobset that is furthest below its scheduling share over a certain time window (currently, the last day). Withing that jobset, it will pick the build with the highest priority.

So meta.schedulingPriority now only determines the order of builds within a jobset, not between jobsets. This makes it much easier to prioritise one jobset over another (e.g. nixpkgs:trunk over nixpkgs:stdenv).

Created by Eelco Dolstra on September 21, 2013

GEADFVZ5LXXFIE3VIP4UJ4AEI2VX57DXER47JA4IHH5BG3QNPAEAC

Dependencies

In channels

upstream

main

Change contents

Replacement in src/lib/Hydra/Controller/Jobset.pm at line 53 [12.104]

B:BD[10.174] → [11.0:118]

    ($c->stash->{latestEval}) = $c->stash->{jobset}->jobsetevals->search({}, { rows => 1, order_by => ["id desc"] });

[10.174]

[13.7208]

    $c->stash->{latestEval} = $c->stash->{jobset}->jobsetevals->search({}, { rows => 1, order_by => ["id desc"] })->single;
    $c->stash->{totalShares} = getTotalShares($c->model('DB')->schema);

Replacement in src/lib/Hydra/Controller/Jobset.pm at line 166 [12.104]

B:BD[14.826] → [14.826:1096]

	foreach my $b (@builds) {
	    my $jobName = $b->get_column('job');
	    $evals->{$eval->id}->{$jobName} = 
	        { id => $b->id, finished => $b->finished, buildstatus => $b->buildstatus };
	    $jobs{$jobName} = 1;
	    $nrBuilds++;
	}
	last if $nrBuilds >= 10000;

[14.826]

[15.1858]

        foreach my $b (@builds) {
            my $jobName = $b->get_column('job');
            $evals->{$eval->id}->{$jobName} = 
                { id => $b->id, finished => $b->finished, buildstatus => $b->buildstatus };
            $jobs{$jobName} = 1;
            $nrBuilds++;
        }
        last if $nrBuilds >= 10000;

Replacement in src/lib/Hydra/Controller/Jobset.pm at line 177 [12.104]

B:BD[7.144] → [7.144:378]

	$c->stash->{showInactive} = 1;
	foreach my $job ($c->stash->{jobset}->jobs->search({ name => { ilike => $filter } })) {
	    next if defined $jobs{$job->name};
	    $c->stash->{inactiveJobs}->{$job->name} = $jobs{$job->name} = 1;
	}

[7.144]

[7.378]

        $c->stash->{showInactive} = 1;
        foreach my $job ($c->stash->{jobset}->jobs->search({ name => { ilike => $filter } })) {
            next if defined $jobs{$job->name};
            $c->stash->{inactiveJobs}->{$job->name} = $jobs{$job->name} = 1;
        }

Insertion in src/lib/Hydra/Controller/Jobset.pm at line 214 [12.104]
[16.705]
[16.705]
```
    $c->stash->{totalShares} = getTotalShares($c->model('DB')->schema);
```

Insertion in src/lib/Hydra/Controller/Jobset.pm at line 293 [12.104]

[9.174]

[16.2272]

        , schedulingshares => int($c->stash->{params}->{schedulingshares})

Insertion in src/lib/Hydra/Controller/Project.pm at line 204 [17.1245]
[16.5027]
[16.5027]
```
    $c->stash->{totalShares} = getTotalShares($c->model('DB')->schema);
```
Replacement in src/lib/Hydra/Helper/Nix.pm at line 23 [19.436]
B:BD[18.1545] → [6.0:35]
```
    captureStdoutStderr run grab);
```
[18.1545]
[20.77]
```
    captureStdoutStderr run grab
    getTotalShares);
```

Insertion in src/lib/Hydra/Helper/Nix.pm at line 537 [19.436]

[6.1010]

[21.850]

sub getTotalShares {
    my ($db) = @_;
    return $db->resultset('Jobsets')->search(
        { 'project.enabled' => 1, 'me.enabled' => 1 },
        { join => 'project', select => { sum => 'schedulingshares' }, as => 'sum' })->single->get_column('sum');
}

Insertion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 17 [8.3643]
[8.3893]
[8.3893]
```
=head1 COMPONENTS LOADED
```

Insertion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 20 [8.3643]

[8.3894]

=over 4
=item * L<Hydra::Component::ToJSON>
=back
=cut
__PACKAGE__->load_components("+Hydra::Component::ToJSON");

Deletion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 39 [8.3643]
B:BD[8.4007] → [8.4007:4060]
```
  data_type: 'text'
  is_nullable: 0
=head2 branch
```
Insertion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 57 [8.3643]
[8.4266]
[8.4266]
```
=head2 revcount
```
Insertion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 60 [8.3643]
[8.4267]
[8.4267]
```
  data_type: 'integer'
  is_nullable: 0
```
Deletion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 70 [8.3643]
B:BD[8.4412] → [8.4412:4474]
```
  "revcount",
  { data_type => "integer", is_nullable => 0 },
```
Insertion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 74 [8.3643]
[8.4595]
[8.4595]
```
  "revcount",
  { data_type => "integer", is_nullable => 0 },
```
Deletion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 84 [8.3643]
B:BD[8.4645] → [8.4645:4665]
```
=item * L</branch>
```

Replacement in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 93 [8.3643]

B:BD[8.4751] → [8.4751:4893]

# Created by DBIx::Class::Schema::Loader v0.07014 @ 2011-12-05 14:15:43
# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:fx3yosWMmJ+MnvL/dSWtFA

[8.4751]

[8.4893]

# Created by DBIx::Class::Schema::Loader v0.07033 @ 2013-09-20 11:08:50
# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:Yl1slt3SAizijgu0KUTn0A

Insertion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 97 [8.3643]

[8.4894]

# You can replace this text with custom code or comments, and it will be preserved on regeneration

Insertion in src/lib/Hydra/Schema/Jobsets.pm at line 119 [24.1960]

[22.226]

[23.256]

  is_nullable: 0
=head2 schedulingshares
  data_type: 'integer'
  default_value: 100

Insertion in src/lib/Hydra/Schema/Jobsets.pm at line 160 [24.1960]

[22.316]

[24.2524]

  "schedulingshares",
  { data_type => "integer", default_value => 100, is_nullable => 0 },

Replacement in src/lib/Hydra/Schema/Jobsets.pm at line 283 [24.1960]

B:BD[24.3136] → [13.31884:32026]

# Created by DBIx::Class::Schema::Loader v0.07033 @ 2013-06-13 01:54:50
# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:tsGR8MhZRIUeNwpcVczMUw

[24.3136]

[25.22627]

# Created by DBIx::Class::Schema::Loader v0.07033 @ 2013-09-20 12:15:23
# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:pD6tGW0Ob3fuA1p0uQnBWw

Insertion in src/root/edit-jobset.tt at line 3 [26.367]
[26.520]
[26.520]
```
[% USE format %]
```
Insertion in src/root/edit-jobset.tt at line 99 [26.367]
[22.860]
[22.860]
```
      <label class="control-label">Scheduling shares</label>
```

Insertion in src/root/edit-jobset.tt at line 101 [26.367]

[22.889]

[26.3705]

        <div class="input-append">
          <input type="number" class="span3" name="schedulingshares" [% HTML.attributes(value => jobset.schedulingshares) %]/>
        </div>
        [% IF totalShares %]
          <span class="help-inline">([% f = format("%.2f"); f(jobset.schedulingshares / totalShares * 100) %]% out of [% totalShares %] shares)</span>
        [% END %]
      </div>
    </div>
    <div class="control-group">
      <div class="controls">

Insertion in src/root/jobset.tt at line 3 [16.8409]
[16.8624]
[16.8841]
```
[% USE format %]
```

Insertion in src/root/jobset.tt at line 126 [16.8409]

[26.9334]

        <th>Scheduling shares:</th>
        <td>[% jobset.schedulingshares %] [% IF totalShares %] ([% f = format("%.2f"); f(jobset.schedulingshares / totalShares * 100) %]% out of [% totalShares %] shares)[% END %]</td>
      </tr>
      <tr>

Replacement in src/script/hydra-queue-runner at line 31 [28.7823]

B:BD[27.116] → [5.16:97]

                if (!defined $lastTime || $build->starttime < $lastTime - 300) {

[27.116]

[27.558]

                if (!defined $lastTime || $build->starttime < $lastTime - 600) {

Replacement in src/script/hydra-queue-runner at line 73 [28.7823]

B:BD[29.113] → [29.113:186]

        foreach my $system (${$machines}{$machineName}{'systemTypes'}) {

[29.113]

[29.186]

        foreach my $system (@{${$machines}{$machineName}{'systemTypes'}}) {

Replacement in src/script/hydra-queue-runner at line 80 [28.7823]

B:BD[30.23] → [4.209:266]

        # Cache scheduled by derivation path to speed up

[30.23]

[4.266]

        # Cache scheduled builds by derivation path to speed up

Replacement in src/script/hydra-queue-runner at line 84 [28.7823]

B:BD[4.379] → [4.379:491]

            foreach $db->resultset('Builds')->search({ finished => 0, enabled => 1 }, { join => ['project'] });

[4.379]

[4.491]

            foreach $db->resultset('Builds')->search({ finished => 0 }, { join => ['project'] });

Replacement in src/script/hydra-queue-runner at line 88 [28.7823]

B:BD[30.139] → [31.350:406]

            { finished => 0, busy => 0, enabled => 1 },

[30.139]

[32.7034]

            { finished => 0, busy => 0 },

Insertion in src/script/hydra-queue-runner at line 91 [28.7823]

[33.37575]

[30.236]

        # Get the total number of scheduling shares.
        my $totalShares = getTotalShares($db);

Replacement in src/script/hydra-queue-runner at line 95 [28.7823]

B:BD[30.303] → [30.303:429]

        # concurrent build for that system type.  Choose the highest
        # priority builds first, then the oldest builds.

[30.303]

[30.429]

        # concurrent build for that system type.

Replacement in src/script/hydra-queue-runner at line 106 [28.7823]
∅:D[34.288] → [30.892:944]
∅:D[29.388] → [30.892:944]
∅:D[35.2781] → [30.892:944]
B:BD[30.892] → [30.892:944]
```
            $extraAllowed = 0 if $extraAllowed < 0;
```
[34.288]
[30.944]
```
            next if $extraAllowed <= 0;
```

Replacement in src/script/hydra-queue-runner at line 108 [28.7823]

B:BD[30.945] → [30.945:1004]

B:BD[30.1004] → [36.540:625]

B:BD[36.625] → [31.407:494]

B:BD[31.494] → [3.0:79]

            # Select the highest-priority builds to start.
            my @builds = $extraAllowed == 0 ? () : $db->resultset('Builds')->search(
                { finished => 0, busy => 0, system => $system->system, enabled => 1 },
                { join => ['project'], order_by => ["priority DESC", "id"] });

[30.945]

[30.1292]

            print STDERR "starting at most $extraAllowed builds for system ${\$system->system}\n";
            j: while ($extraAllowed-- > 0) {

Replacement in src/script/hydra-queue-runner at line 112 [28.7823]

B:BD[30.1293] → [3.80:109]

∅:D[3.109] → [36.686:728]

B:BD[28.8821] → [36.686:728]

B:BD[36.728] → [37.428:758]

B:BD[37.758] → [4.493:578]

∅:D[4.578] → [37.829:864]

B:BD[37.829] → [37.829:864]

            my $started = 0;
            foreach my $build (@builds) {
                # Find a dependency of $build that has no queued
                # dependencies itself.  This isn't strictly necessary,
                # but it ensures that Nix builds are done as part of
                # their corresponding Hydra builds, rather than as a
                # dependency of some other Hydra build.
                while (my $dep = findBuildDependencyInQueue($buildsByDrv, $build)) {
                    $build = $dep;

[30.1293]

[37.864]

                my @runnableJobsets = $db->resultset('Builds')->search(
                    { finished => 0, busy => 0, system => $system->system },
                    { select => ['project', 'jobset'], distinct => 1 });
                next if @runnableJobsets == 0;
                my $windowSize = 24 * 3600;
                my $totalWindowSize = $windowSize * $max;
                my @res;
                foreach my $b (@runnableJobsets) {
                    my $jobset = $db->resultset('Jobsets')->find($b->get_column('project'), $b->get_column('jobset')) or die;
                    my $duration = $jobset->builds->search(
                        { },
                        { where => \ ("(finished = 0 or (me.stoptime >= " . (time() - $windowSize) . "))")
                        , join => 'buildsteps'
                        , select => \ "sum(coalesce(buildsteps.stoptime, ${\time}) - buildsteps.starttime)"
                        , as => "sum" })->single->get_column("sum") // 0;
                    # Add a 30s penalty for each started build.  This
                    # is to account for jobsets that have running
                    # builds but no build steps yet.
                    $duration += $jobset->builds->search({ finished => 0, busy => 1 })->count * 30;
                    my $share = $jobset->schedulingshares;
                    my $delta = ($share / $totalShares) - ($duration / $totalWindowSize);
                    #printf STDERR "%s:%s: %d s, %.3f%%, allowance = %.3f%%\n", $jobset->get_column('project'), $jobset->name, $duration, $duration / $totalWindowSize, $delta;
                    push @res, { jobset => $jobset, delta => $delta };

Replacement in src/script/hydra-queue-runner at line 145 [28.7823]

B:BD[37.882] → [37.882:920]

                next if $build->busy;

[37.882]

[38.644]


                foreach my $r (sort { $b->{delta} <=> $a->{delta} } @res) {
                    my $jobset = $r->{jobset};
                    #print STDERR "selected ", $jobset->get_column('project'), ':', $jobset->name, "\n";

Replacement in src/script/hydra-queue-runner at line 150 [28.7823]

∅:D[38.645] → [36.728:790]

B:BD[36.728] → [36.728:790]

B:BD[36.790] → [21.1310:1351]

∅:D[36.790] → [39.60:94]

∅:D[21.1351] → [39.60:94]

B:BD[39.60] → [39.60:94]

B:BD[39.94] → [40.10908:11115]

∅:D[32.7477] → [36.1104:1149]

∅:D[40.11115] → [36.1104:1149]

B:BD[36.1104] → [36.1104:1149]

                my $logfile = getcwd . "/logs/" . $build->id;
                mkdir(dirname $logfile);
                unlink($logfile);
                $build->update(
                    { busy => 1
                    , locker => $$
                    , logfile => $logfile
                    , starttime => time()
                    });
                push @buildsStarted, $build;

[38.645]

[3.110]

                    # Select the highest-priority build for this jobset.
                    my @builds = $jobset->builds->search(
                        { finished => 0, busy => 0, system => $system->system },
                        { order_by => ["priority DESC", "id"] });
                    foreach my $build (@builds) {
                        # Find a dependency of $build that has no queued
                        # dependencies itself.  This isn't strictly necessary,
                        # but it ensures that Nix builds are done as part of
                        # their corresponding Hydra builds, rather than as a
                        # dependency of some other Hydra build.
                        while (my $dep = findBuildDependencyInQueue($buildsByDrv, $build)) {
                            $build = $dep;
                        }
                        next if $build->busy;
                        printf STDERR "starting build %d (%s:%s:%s) on %s (jobset allowance = %.3f%%)\n", 
                            $build->id, $build->project->name, $build->jobset->name, $build->job->name, $build->system, $r->{delta};

Replacement in src/script/hydra-queue-runner at line 169 [28.7823]

B:BD[3.111] → [3.111:178]

                last if ++$started >= $extraAllowed;
            }

[3.111]

[3.178]

                        my $logfile = getcwd . "/logs/" . $build->id;
                        mkdir(dirname $logfile);
                        unlink($logfile);
                        $build->update(
                            { busy => 1
                            , locker => $$
                            , logfile => $logfile
                            , starttime => time()
                            });
                        push @buildsStarted, $build;
                        next j;
                    }
                }

Replacement in src/script/hydra-queue-runner at line 183 [28.7823]

B:BD[3.179] → [3.179:358]

            if ($started > 0) {
                print STDERR "system type `", $system->system,
                    "': $nrActive active, $max allowed, started $started builds\n";

[3.179]

[30.1879]

                last; # nothing found, give up on this system type

Deletion in src/script/hydra-queue-runner at line 192 [28.7823]

B:BD[36.1220] → [2.0:143]

        print "starting build $id (", $build->project->name, ":", $build->jobset->name, ':', $build->job->name, ") on ", $build->system, "\n";

Insertion in src/sql/hydra.sql at line 64 [42.1]
[22.1382]
[41.5479]
```
    schedulingShares integer not null default 100,
```

File addition: upgrade-21.sql (----------)

[43.3004]

alter table Jobsets
    add column schedulingShares integer not null default 100;

hydra-queue-runner: Improved scheduling

Dependencies

In channels

Change contents

Replacement in src/lib/Hydra/Controller/Jobset.pm at line 53 [12.104]

Replacement in src/lib/Hydra/Controller/Jobset.pm at line 166 [12.104]

Replacement in src/lib/Hydra/Controller/Jobset.pm at line 177 [12.104]

Insertion in src/lib/Hydra/Controller/Jobset.pm at line 214 [12.104]

Insertion in src/lib/Hydra/Controller/Jobset.pm at line 293 [12.104]

Insertion in src/lib/Hydra/Controller/Project.pm at line 204 [17.1245]

Replacement in src/lib/Hydra/Helper/Nix.pm at line 23 [19.436]

Insertion in src/lib/Hydra/Helper/Nix.pm at line 537 [19.436]

Insertion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 17 [8.3643]

Insertion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 20 [8.3643]

Deletion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 39 [8.3643]

Insertion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 57 [8.3643]

Insertion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 60 [8.3643]

Deletion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 70 [8.3643]

Insertion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 74 [8.3643]

Deletion in src/lib/Hydra/Schema/CachedDarcsInputs.pm at line 84 [8.3643]