Allow only 1 thread to send a closure to a given machine at the same time

Dependencies

• [2] ED4M6GSC Typo
• [3] HJOEIMLR Refactor
• [4] N4IROACV Move buildRemote() into State
• [5] NJJ7H64S Very basic multi-threaded queue runner
• [6] 7VQ4ALFY Update "make check" for the new queue runner
• [7] A2GL5FOZ Moar stats
• [8] OG3Z3QGC Namespace cleanup
• [9] 24BMQDZA Start of single-process hydra-queue-runner
• [10] 6TY4LNHH Finish copyClosure
• [11] 5AIYUMTB Basic remote building
• [12] OCZ4LSGG Automatically retry aborted builds
• [13] LE4VZIY5 More stats
• [14] MB3TISH2 Rate-limit the number of threads copying closures at the same time

Change contents

• replacement in src/hydra-queue-runner/build-remote.cc at line 73
  [4.1406]→[4.0:63](∅→∅)
  − TokenServer & copyClosureTokenServer, counter & bytesSent,
  [4.1406]

  [4.1406]
  + counter & bytesSent,
• edit in src/hydra-queue-runner/build-remote.cc at line 101
  [4.284]→[4.2295:2296](∅→∅),[4.2295]→[4.2295:2296](∅→∅),[4.2296]→[4.43:398](∅→∅)
  −

  − /* Ensure that only a limited number of threads can copy closures

  − at the same time. However, proceed anyway after a timeout to

  − prevent starvation by a handful of really huge closures. */

  − time_t start = time(0);

  − int timeout = 60 * (10 + rand() % 5);

  − auto token(copyClosureTokenServer.get(timeout));

  − time_t stop = time(0);
• edit in src/hydra-queue-runner/build-remote.cc at line 102
  [4.399]→[4.399:515](∅→∅),[4.515]→[2.0:98](∅→∅),[2.98]→[4.613:614](∅→∅),[4.613]→[4.613:614](∅→∅)
  − if (token())

  − printMsg(lvlDebug, format("got copy closure token after %1%s") % (stop - start));

  − else

  − printMsg(lvlDebug, format("did not get copy closure token after %1%s") % (stop - start));

  −
• replacement in src/hydra-queue-runner/build-remote.cc at line 184
  [4.205]→[4.595:678](∅→∅)
  − copyClosureTo(store, from, to, inputs, copyClosureTokenServer, bytesSent);
  [4.205]

  [4.277]
  + std::lock_guard<std::mutex> sendLock(machine->state->sendLock);

  + copyClosureTo(store, from, to, inputs, bytesSent);
• edit in src/hydra-queue-runner/hydra-queue-runner.cc at line 35
  [4.7501]→[3.1073:1126](∅→∅)
  − : copyClosureTokenServer{maxParallelCopyClosure}
• edit in src/hydra-queue-runner/state.hh at line 15
  [3.1361]→[3.1361:1388](∅→∅)
  − #include "token-server.hh"
• edit in src/hydra-queue-runner/state.hh at line 138
  [3.3777]

  [3.3777]
  +

  + /* Mutex to prevent multiple threads from sending data to the

  + same machine (which would be inefficient). */

  + std::mutex sendLock;
• edit in src/hydra-queue-runner/state.hh at line 197
  [3.5335]→[3.5335:5497](∅→∅)
  − /* Token server limiting the number of threads copying closures in

  − parallel to prevent excessive I/O load. */

  − TokenServer copyClosureTokenServer;

  −