grahamc/hydra2 - Change S2NXJOJE3V7PTXHA2HAHA5BUSNJRHZNXBWK7KSWTPGISWKXYMJDQC

Fix a race that can cause hydra-queue-runner to ignore newly added builds

As @dtzWill discovered, with the concurrent hydra-evaluator, there can be multiple active transactions adding builds to the database. As a result, builds can become visible in a non-monotonically increasing order, breaking the queue monitor's assumption that build IDs only go up.

The fix is to have hydra-eval-jobset provide the lowest build ID it just added in the builds_added notification, and have the queue monitor check from there.

Fixes #496.

Created by Eelco Dolstra on July 21, 2017

S2NXJOJE3V7PTXHA2HAHA5BUSNJRHZNXBWK7KSWTPGISWKXYMJDQC

Dependencies

In channels

main

Change contents

Replacement in src/hydra-queue-runner/queue-monitor.cc at line 51 [4.20067]

B:BD[4.21027] → [4.21027:21058]

        if (buildsAdded.get())

[4.21027]

[4.21058]

        if (auto lowestId = buildsAdded.get()) {
            lastBuildId = std::stoi(*lowestId) - 1;

Insertion in src/hydra-queue-runner/queue-monitor.cc at line 54 [4.20067]
[4.21145]
[4.21145]
```
        }
```

Replacement in src/hydra-queue-runner/queue-monitor.cc at line 80 [4.20067]

B:BD[4.21622] → [4.21622:21709]

    printMsg(lvlInfo, format("checking the queue for builds > %1%...") % lastBuildId);

[4.21622]

[4.21709]

    printInfo("checking the queue for builds > %d...", lastBuildId);

Replacement in src/libhydra/db.hh at line 24 [5.93]

B:BD[5.626] → [5.626:679]

struct receiver : public pqxx::notification_receiver

[5.626]

[5.679]

class receiver : public pqxx::notification_receiver

Replacement in src/libhydra/db.hh at line 26 [5.93]

B:BD[5.681] → [5.681:706]

    bool status = false;

[5.681]

[5.706]

    std::experimental::optional<std::string> status;
public:

Insertion in src/libhydra/db.hh at line 32 [5.93]
[5.829]
[2.216]
Replacement in src/libhydra/db.hh at line 35 [5.93]
B:BD[5.898] → [5.898:921]
```
        status = true;
```
[5.898]
[5.921]
```
        status = payload;
```

Replacement in src/libhydra/db.hh at line 37 [5.93]

B:BD[5.928] → [5.928:1012]

    bool get() {
        bool b = status;
        status = false;
        return b;

[5.928]

[5.1012]


    std::experimental::optional<std::string> get() {
        auto s = status;
        status = std::experimental::nullopt;
        return s;

Insertion in src/script/hydra-eval-jobset at line 28 [8.1319]
[6.745]
[7.961]
```
my $notifyAdded = $db->storage->dbh->prepare("notify builds_added, ?");
```

Insertion in src/script/hydra-eval-jobset at line 725 [8.1319]

[9.51]

[10.518]


            # Wake up hydra-queue-runner.
            my $lowestId;
            while (my ($id, $x) = each %buildMap) {
                $lowestId = $id if $x->{new} && (!defined $lowestId || $id < $lowestId);
            }
            $notifyAdded->execute($lowestId) if defined $lowestId;

Deletion in src/sql/hydra.sql at line 236 [8.1]

B:BD[3.1] → [3.1:210]

∅:D[3.210] → [11.135:136]

B:BD[11.135] → [11.135:136]

create function notifyBuildsAdded() returns trigger as 'begin notify builds_added; return null; end;' language plpgsql;
create trigger BuildsAdded after insert on Builds execute procedure notifyBuildsAdded();

File addition: upgrade-54.sql (----------)

[12.3004]

drop trigger BuildsAdded on Builds;
drop function notifyBuildsAdded();