grahamc/hydra2 - Change UANT7MOVK62YKFDQ5AJU2XETFKPUAEMJ4WJPRNJ7OJXN2E6DOFLQC

hydra-evaluator: add a 'ONE_AT_A_TIME' evaluator style

In the past, jobsets which are automatically evaluated are evaluated regularly, on a schedule. This schedule means a new evaluation is created every checkInterval seconds (assuming something changed.)

This model works well for architectures where our build farm can easily keep up with demand.

This commit adds a new type of evaluation, called ONE_AT_A_TIME, which only schedules a new evaluation if the previous evaluation of the jobset has no unfinished builds.

This model of evaluation lets us have 'low-tier' architectures.

For example, we could now have a jobset for ARMv7l builds, where the buildfarm only has a single, underpowered ARMv7l builder. Configuring that jobset as ONE_AT_A_TIME will create an evaluation and then won't schedule another evaluation until every job of the existing evaluation is complete.

This way, the cache will have a complete collection of pre-built software for some commits, but the underpowered architecture will never become backlogged in ancient revisions.

Created by Graham Christensen on March 3, 2020

UANT7MOVK62YKFDQ5AJU2XETFKPUAEMJ4WJPRNJ7OJXN2E6DOFLQC

Dependencies

In channels

main

Change contents

Insertion in src/hydra-evaluator/hydra-evaluator.cc at line 18 [4.575]

[3.166]

[4.761]

enum class EvaluationStyle
{
    SCHEDULE = 1,
    ONESHOT = 2,
    ONE_AT_A_TIME = 3,
};

Insertion in src/hydra-evaluator/hydra-evaluator.cc at line 34 [4.575]
[4.925]
[4.925]
```
        std::optional<EvaluationStyle> evaluation_style;
```

Replacement in src/hydra-evaluator/hydra-evaluator.cc at line 71 [4.575]

B:BD[4.1527] → [4.1527:1664]

            ("select project, j.name, lastCheckedTime, triggerTime, checkInterval from Jobsets j join Projects p on j.project = p.name "

[4.1527]

[4.1664]

            ("select project, j.name, lastCheckedTime, triggerTime, checkInterval, j.enabled as jobset_enabled from Jobsets j join Projects p on j.project = p.name "

Insertion in src/hydra-evaluator/hydra-evaluator.cc at line 89 [4.575]

[4.2284]

            switch (row["jobset_enabled"].as<int>(0)) {
                case 1:
                    jobset.evaluation_style = EvaluationStyle::SCHEDULE;
                    break;
                case 2:
                    jobset.evaluation_style = EvaluationStyle::ONESHOT;
                    break;
                case 3:
                    jobset.evaluation_style = EvaluationStyle::ONE_AT_A_TIME;
                    break;
            }

Insertion in src/hydra-evaluator/hydra-evaluator.cc at line 155 [4.575]

[2.111]

            debug("shouldEvaluate %s:%s? no: already running",
                  jobset.name.first, jobset.name.second);

Replacement in src/hydra-evaluator/hydra-evaluator.cc at line 160 [4.575]

B:BD[2.148] → [2.148:220]

        if (jobset.triggerTime == std::numeric_limits<time_t>::max()) {

[2.148]

[2.220]

        if (jobset.triggerTime != std::numeric_limits<time_t>::max()) {

Insertion in src/hydra-evaluator/hydra-evaluator.cc at line 162 [4.575]

[2.277]

            debug("shouldEvaluate %s:%s? yes: requested",
                  jobset.name.first, jobset.name.second);

Insertion in src/hydra-evaluator/hydra-evaluator.cc at line 170 [4.575]

[2.480]

            debug("shouldEvaluate %s:%s? no: checkInterval <= 0",
                  jobset.name.first, jobset.name.second);

Insertion in src/hydra-evaluator/hydra-evaluator.cc at line 174 [4.575]

[2.516]


        if (jobset.lastCheckedTime + jobset.checkInterval <= time(0)) {
            // Time to schedule a fresh evaluation. If the jobset
            // is a ONE_AT_A_TIME jobset, ensure the previous jobset
            // has no remaining, unfinished work.
            auto conn(dbPool.get());

Insertion in src/hydra-evaluator/hydra-evaluator.cc at line 182 [4.575]
[2.517]
[2.517]
```
            pqxx::work txn(*conn);
```

Replacement in src/hydra-evaluator/hydra-evaluator.cc at line 184 [4.575]

B:BD[2.518] → [2.518:666]

        if (jobset.lastCheckedTime + jobset.checkInterval <= time(0)) {
            // Time to schedule a fresh evaluation
            return true;

[2.518]

[2.666]

            if (jobset.evaluation_style == EvaluationStyle::ONE_AT_A_TIME) {
                auto evaluation_res = txn.parameterized
                    ("select id from JobsetEvals "
                     "where project = $1 and jobset = $2 "
                     "order by id desc limit 1")
                  (jobset.name.first)
                  (jobset.name.second)
                  .exec();
                if (evaluation_res.empty()) {
                    // First evaluation, so allow scheduling.
                    debug("shouldEvaluate(one-at-a-time) %s:%s? yes: no prior eval",
                          jobset.name.first, jobset.name.second);
                    return true;
                }
                auto evaluation_id = evaluation_res[0][0].as<int>();
                auto unfinished_build_res = txn.parameterized
                    ("select id from Builds "
                     "join JobsetEvalMembers "
                     "    on (JobsetEvalMembers.build = Builds.id) "
                     "where JobsetEvalMembers.eval = $1 "
                     "  and builds.finished = 0 "
                     " limit 1")
                  (evaluation_id)
                  .exec();
                // If the previous evaluation has no unfinished builds
                // schedule!
                if (unfinished_build_res.empty()) {
                    debug("shouldEvaluate(one-at-a-time) %s:%s? yes: no unfinished builds",
                          jobset.name.first, jobset.name.second);
                    return true;
                } else {
                    debug("shouldEvaluate(one-at-a-time) %s:%s? no: at least one unfinished build",
                           jobset.name.first, jobset.name.second);
                    return false;
                }
            } else {
                // EvaluationStyle::ONESHOT, EvaluationStyle::SCHEDULED
                debug("shouldEvaluate(oneshot/scheduled) %s:%s? yes: checkInterval elapsed",
                      jobset.name.first, jobset.name.second);
                return true;
            }

Replacement in src/lib/Hydra/Controller/Jobset.pm at line 229 [6.104]
B:BD[5.108] → [5.108:149]
```
    die if $enabled < 0 || $enabled > 2;
```
[5.108]
[7.2074]
```
    die if $enabled < 0 || $enabled > 3;
```

Insertion in src/root/edit-jobset.tt at line 71 [8.367]

[5.684]

          <button type="button" class="btn" value="3">One-at-a-time</button>

Replacement in src/root/jobset.tt at line 132 [7.8409]

B:BD[5.852] → [5.852:992]

        <td>[% IF jobset.enabled == 0; "Disabled"; ELSIF jobset.enabled == 1; "Enabled"; ELSIF jobset.enabled == 2; "One-shot"; END %]</td>

[5.852]

[5.992]

        <td>[% IF jobset.enabled == 0; "Disabled"; ELSIF jobset.enabled == 1; "Enabled"; ELSIF jobset.enabled == 2; "One-shot"; ELSIF jobset.enabled == 3; "One-at-a-time"; END %]</td>

Replacement in src/sql/hydra.sql at line 64 [10.1]

B:BD[9.3384] → [5.1675:1764]

    enabled       integer not null default 1, -- 0 = disabled, 1 = enabled, 2 = one-shot

[9.3384]

[11.5089]

    enabled       integer not null default 1, -- 0 = disabled, 1 = enabled, 2 = one-shot, 3 = one-at-a-time