{"header":{"message":"hydra-queue-runner: Limit memory usage","description":"When using a binary cache store, the queue runner receives NARs from\nthe build machines, compresses them, and uploads them to the\ncache. However, keeping multiple large NARs in memory can cause the\nqueue runner to run out of memory. This can happen for instance when\nit's processing multiple ISO images concurrently.\n\nThe fix is to use a TokenServer to prevent the builder threads to\nstore more than a certain total size of NARs concurrently (at the\nmoment, this is hard-coded at 4 GiB). Builder threads that cause the\nlimit to be exceeded will block until other threads have finished.\n\nThe 4 GiB limit does not include certain other allocations, such as\nfor xz compression or for FSAccessor::readFile(). But since these are\nunlikely to be more than the size of the NARs and hydra.nixos.org has\n32 GiB RAM, it should be fine.","timestamp":"2016-03-09T13:30:13Z","authors":[{"email":"eelco.dolstra@logicblox.com","name":"Eelco Dolstra"}]},"hash":"SL3WSRACCX2IMJHHLTRAUQT7QDLCOKYLVO2FEHWIHXM5GPKSRJTQC","authors":[],"deps":{"hashes":[["SL3WSRACCX2IMJHHLTRAUQT7QDLCOKYLVO2FEHWIHXM5GPKSRJTQC","hydra-queue-runner: Limit memory usage"],["AA",null],["B2L4T3X63XVYJQXEDU4WT5Y4R6PMDXGC6WN2KGOMHRQILSABNQOAC","Sync with Nix"],["UVNTWTWGQOFKDAJ2ROJYT4U2N4EUXKNWZWPHOM42WPLUL4ALXRJQC","Prevent download of NARs we just uploaded"],["BRAESISHTN4IIWUBVDMPDMY7QLMJDKX7GQ7K6NSJN66L5VPWSX3QC","Warn if PostgreSQL appears stalled"],["MB3TISH2KYBIGY6XJKMN4HO2S6TCN2GORJENMECCKLXGGIRS2O2AC","Rate-limit the number of threads copying closures at the same time"],["GJV2J5HXFKVF7BXNFMTI6ZZMYADXIKTFIQJ3FONJ7DPVCUJZ2L4AC","Pool local store connections"],["HJOEIMLRDVQ2KZI5HGL2HKGBM3AHP7YIKGKDAGFUNKRUXVRB24NAC","Refactor"],["5AIYUMTBY6TFQTBRP3MJ2PYWUMRF57I77NIVWYE74UMEVQMBWZVQC","Basic remote building"],["A2GL5FOZ3UJ2NM5RPRWTNPFTKLBA54B2UC6UIYO4M3N3RFNC4BTAC","Moar stats"],["FITVNQ2SVM6KSOF5P3HHWJYQ3WMQYDJGAONCBIZ7OF7CPXGMA36QC","Keep track of the time we spend copying to/from build machines"],["CNLNT3T4IXU42GAC64CU5CEDTDLM2SHSJZEVHTMOCWYFXSQNC76QC","Allow only 1 thread to send a closure to a given machine at the same time"],["73YR46NJNYZQKHA3QDJCAZYAKC2CGEF5LIS44NOIPDZU6FX6BDPQC","hydra-queue-runner: Write directly to a binary cache"],["6EO3HVNAAVJLBYTBSHXQADDTYF7JJU3ANHS5SKHSSROOW4AXIZ4AC","Merge remote-tracking branch 'origin/master' into binary-cache"],["OG3Z3QGCG2FNDGF4VQYWOXLZCF7LGTZCUMGKUCBNAPAQ5FRFHQ2AC","Namespace cleanup"],["LE4VZIY5VZ52FOP5QQRIJINWIMWTAPRTZTGO77JXUEPGRPRSQYMAC","More stats"],["DIEY5USNXFN6O2ARZ5IPBS2VX4OH7KE4RMX2JWAWTVMMSYA6GREAC","Keep better bytesReceived/bytesSent stats"],["N4IROACVZ4MU73J5SM6WXJMKQSFR3VN5SOKENNNZNEGMTGB2Q3HAC","Move buildRemote() into State"],["24BMQDZAWDQ7VNIA7TIROXSOYLOJBNZ2E4264WHWNJAEN6ZB3UOAC","Start of single-process hydra-queue-runner"],["PLOZBRTR6USSGJX7GR2RZKNPVYG2Q6QM7LW6IA35MKL63ZTQVD7QC","Add command ‘hydra-queue-runner --status’ to show current status"],["IK2UBDAU6QKUXHJG3SXJKYGIIXRDKI6UVRTFC6ZVDXDCGNCMEWVAC","Revive jobset scheduling"],["EYR3EW6JVHNVLXMI57FUVPHQAHPETBML4H44OGJFHUT54KTTHIGQC","Keep stats for the Hydra auto scaler"],["EOO4EFWD2BJCGF3ZKS2QR3XDW4WHUGH2EHSOFVK6GMI5BUBZW6QQC","Use a single BinaryCacheStore for all threads"]],"deps":["B2L4T3X63XVYJQXEDU4WT5Y4R6PMDXGC6WN2KGOMHRQILSABNQOAC","UVNTWTWGQOFKDAJ2ROJYT4U2N4EUXKNWZWPHOM42WPLUL4ALXRJQC","BRAESISHTN4IIWUBVDMPDMY7QLMJDKX7GQ7K6NSJN66L5VPWSX3QC"],"known":["5AIYUMTBY6TFQTBRP3MJ2PYWUMRF57I77NIVWYE74UMEVQMBWZVQC","MB3TISH2KYBIGY6XJKMN4HO2S6TCN2GORJENMECCKLXGGIRS2O2AC","LE4VZIY5VZ52FOP5QQRIJINWIMWTAPRTZTGO77JXUEPGRPRSQYMAC","A2GL5FOZ3UJ2NM5RPRWTNPFTKLBA54B2UC6UIYO4M3N3RFNC4BTAC","HJOEIMLRDVQ2KZI5HGL2HKGBM3AHP7YIKGKDAGFUNKRUXVRB24NAC","N4IROACVZ4MU73J5SM6WXJMKQSFR3VN5SOKENNNZNEGMTGB2Q3HAC","OG3Z3QGCG2FNDGF4VQYWOXLZCF7LGTZCUMGKUCBNAPAQ5FRFHQ2AC","CNLNT3T4IXU42GAC64CU5CEDTDLM2SHSJZEVHTMOCWYFXSQNC76QC","FITVNQ2SVM6KSOF5P3HHWJYQ3WMQYDJGAONCBIZ7OF7CPXGMA36QC","73YR46NJNYZQKHA3QDJCAZYAKC2CGEF5LIS44NOIPDZU6FX6BDPQC","6EO3HVNAAVJLBYTBSHXQADDTYF7JJU3ANHS5SKHSSROOW4AXIZ4AC","GJV2J5HXFKVF7BXNFMTI6ZZMYADXIKTFIQJ3FONJ7DPVCUJZ2L4AC","DIEY5USNXFN6O2ARZ5IPBS2VX4OH7KE4RMX2JWAWTVMMSYA6GREAC"]},"hunks":[{"Replacement":{"inode":{"change":5,"pos":85},"path":"src/hydra-queue-runner/build-remote.cc","line":286,"ins":{"up_context":[{"change":5,"pos":1035}],"down_context":[{"change":5,"pos":816}],"flag":"BLOCK","start":0,"end":102,"inode":{"change":5,"pos":85}},"del":{"edges":[{"previous":"","flag":"DELETED","from":{"change":5,"pos":1035},"to":{"change":5,"start":1286,"end":1404},"introduced_by":5},{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":1286},"to":{"change":5,"start":1286,"end":1404},"introduced_by":5}],"inode":{"change":5,"pos":85}},"ins_contents":[{"Add":" MaintainCount mc(nrStepsCopyingFrom);"},{"Add":""},{"Add":" auto now1 = std::chrono::steady_clock::now();"},{"Add":""}],"del_contents":[{"Del":" printMsg(lvlDebug, format(\"copying outputs of ‘%1%’ from ‘%2%’\") % step->drvPath % machine->sshName);\n"}]}},{"Replacement":{"inode":{"change":5,"pos":85},"path":"src/hydra-queue-runner/build-remote.cc","line":293,"ins":{"up_context":[{"change":5,"pos":931}],"down_context":[{"change":5,"pos":217}],"flag":"BLOCK","start":103,"end":1014,"inode":{"change":5,"pos":85}},"del":{"edges":[{"previous":"","flag":"DELETED","from":{"change":5,"pos":931},"to":{"change":5,"start":290,"end":336},"introduced_by":5},{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":290},"to":{"change":5,"start":290,"end":336},"introduced_by":5}],"inode":{"change":5,"pos":85}},"ins_contents":[{"Add":""},{"Add":" /* Query the size of the output paths. */"},{"Add":" size_t totalNarSize = 0;"},{"Add":" to << cmdQueryPathInfos << outputs;"},{"Add":" to.flush();"},{"Add":" while (true) {"},{"Add":" if (readString(from) == \"\") break;"},{"Add":" readString(from); // deriver"},{"Add":" readStrings(from); // references"},{"Add":" readLongLong(from); // download size"},{"Add":" totalNarSize += readLongLong(from);"},{"Add":" }"},{"Add":""},{"Add":" printMsg(lvlDebug, format(\"copying outputs of ‘%s’ from ‘%s’ (%d bytes)\")"},{"Add":" % step->drvPath % machine->sshName % totalNarSize);"},{"Add":""},{"Add":" /* Block until we have the required amount of memory"},{"Add":" available. FIXME: only need this for binary cache"},{"Add":" destination stores. */"},{"Add":" auto resStart = std::chrono::steady_clock::now();"},{"Add":" auto memoryReservation(memoryTokens.get(totalNarSize));"},{"Add":" auto resStop = std::chrono::steady_clock::now();"}],"del_contents":[{"Del":" MaintainCount mc(nrStepsCopyingFrom);\n"}]}},{"Replacement":{"inode":{"change":5,"pos":85},"path":"src/hydra-queue-runner/build-remote.cc","line":316,"ins":{"up_context":[{"change":5,"pos":218}],"down_context":[{"change":3,"pos":231}],"flag":"BLOCK","start":1015,"end":1299,"inode":{"change":5,"pos":85}},"del":{"edges":[{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":218},"to":{"change":3,"start":177,"end":231},"introduced_by":3}],"inode":{"change":5,"pos":85}},"ins_contents":[{"Add":" auto resMs = std::chrono::duration_cast(resStop - resStart).count();"},{"Add":" if (resMs >= 1000)"},{"Add":" printMsg(lvlError, format(\"warning: had to wait %d ms for %d memory tokens for %s\")"},{"Add":" % resMs % totalNarSize % step->drvPath);"}],"del_contents":[{"Del":" result.accessor = destStore->getFSAccessor();\n"}]}},{"Replacement":{"inode":{"change":5,"pos":85},"path":"src/hydra-queue-runner/build-remote.cc","line":321,"ins":{"up_context":[{"change":3,"pos":232}],"down_context":[{"change":5,"pos":272}],"flag":"BLOCK","start":1300,"end":1354,"inode":{"change":5,"pos":85}},"del":{"edges":[{"previous":"","flag":"DELETED","from":{"change":3,"pos":232},"to":{"change":5,"start":218,"end":272},"introduced_by":3},{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":218},"to":{"change":5,"start":218,"end":272},"introduced_by":5}],"inode":{"change":5,"pos":85}},"ins_contents":[{"Add":" result.accessor = destStore->getFSAccessor();"}],"del_contents":[{"Del":" auto now1 = std::chrono::steady_clock::now();\n"}]}},{"Edit":{"inode":{"change":19,"pos":4840},"path":"src/hydra-queue-runner/hydra-queue-runner.cc","line":20,"atom":{"NewVertex":{"up_context":[{"change":19,"pos":7501}],"down_context":[{"change":19,"pos":7501}],"flag":"BLOCK","start":1355,"end":1419,"inode":{"change":19,"pos":4840}}},"contents":[{"Add":" : memoryTokens(4ULL << 30) // FIXME: make this configurable"}]}},{"Edit":{"inode":{"change":19,"pos":4840},"path":"src/hydra-queue-runner/hydra-queue-runner.cc","line":571,"atom":{"NewVertex":{"up_context":[{"change":4,"pos":626}],"down_context":[{"change":20,"pos":1628}],"flag":"BLOCK","start":1420,"end":1488,"inode":{"change":19,"pos":4840}}},"contents":[{"Add":" root.attr(\"memoryTokensInUse\", memoryTokens.currentUse());"},{"Add":""}]}},{"Edit":{"inode":{"change":19,"pos":4840},"path":"src/hydra-queue-runner/hydra-queue-runner.cc","line":595,"atom":{"NewVertex":{"up_context":[{"change":20,"pos":2016}],"down_context":[{"change":21,"pos":1730}],"flag":"BLOCK","start":1489,"end":1490,"inode":{"change":19,"pos":4840}}},"contents":[{"Add":""}]}},{"Edit":{"inode":{"change":19,"pos":4840},"path":"src/hydra-queue-runner/hydra-queue-runner.cc","line":607,"atom":{"NewVertex":{"up_context":[{"change":21,"pos":2189}],"down_context":[{"change":22,"pos":1936}],"flag":"BLOCK","start":1491,"end":1492,"inode":{"change":19,"pos":4840}}},"contents":[{"Add":""}]}},{"Edit":{"inode":{"change":5,"pos":1128},"path":"src/hydra-queue-runner/state.hh","line":12,"atom":{"NewVertex":{"up_context":[{"change":5,"pos":1299}],"down_context":[{"change":5,"pos":1299}],"flag":"BLOCK","start":1493,"end":1547,"inode":{"change":5,"pos":1128}}},"contents":[{"Add":"#include \"token-server.hh\""},{"Add":""},{"Add":"#include \"derivations.hh\""}]}},{"Edit":{"inode":{"change":5,"pos":1128},"path":"src/hydra-queue-runner/state.hh","line":17,"atom":{"EdgeMap":{"edges":[{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":1342},"to":{"change":5,"start":1342,"end":1361},"introduced_by":5},{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":1388},"to":{"change":5,"start":1388,"end":1389},"introduced_by":5}],"inode":{"change":5,"pos":1128}}},"contents":[{"Del":"#include \"sync.hh\"\n"},{"Del":"\n"}]}},{"Replacement":{"inode":{"change":5,"pos":1128},"path":"src/hydra-queue-runner/state.hh","line":18,"ins":{"up_context":[{"change":5,"pos":1413}],"down_context":[{"change":5,"pos":1461}],"flag":"BLOCK","start":1548,"end":1567,"inode":{"change":5,"pos":1128}},"del":{"edges":[{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":1413},"to":{"change":5,"start":1413,"end":1439},"introduced_by":5},{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":1439},"to":{"change":5,"start":1218,"end":1261},"introduced_by":5}],"inode":{"change":5,"pos":1128}},"ins_contents":[{"Add":"#include \"sync.hh\""}],"del_contents":[{"Del":"#include \"derivations.hh\"\n"},{"Del":"\n#include \"binary-cache-store.hh\" // FIXME\n"}]}},{"Edit":{"inode":{"change":5,"pos":1128},"path":"src/hydra-queue-runner/state.hh","line":355,"atom":{"NewVertex":{"up_context":[{"change":23,"pos":1502}],"down_context":[{"change":23,"pos":1502}],"flag":"BLOCK","start":1568,"end":1906,"inode":{"change":5,"pos":1128}}},"contents":[{"Add":""},{"Add":" /* Token server to prevent threads from allocating too many big"},{"Add":" strings concurrently while importing NARs from the build"},{"Add":" machines. When a thread imports a NAR of size N, it will first"},{"Add":" acquire N memory tokens, causing it to block until that many"},{"Add":" tokens are available. */"},{"Add":" nix::TokenServer memoryTokens;"}]}},{"Edit":{"inode":{"change":5,"pos":1640},"path":"src/hydra-queue-runner/token-server.hh","line":6,"atom":{"NewVertex":{"up_context":[{"change":5,"pos":1693}],"down_context":[{"change":5,"pos":1693}],"flag":"BLOCK","start":1907,"end":1944,"inode":{"change":5,"pos":1640}}},"contents":[{"Add":"#include \"types.hh\""},{"Add":""},{"Add":"namespace nix {"}]}},{"Edit":{"inode":{"change":5,"pos":1640},"path":"src/hydra-queue-runner/token-server.hh","line":10,"atom":{"NewVertex":{"up_context":[{"change":5,"pos":1694}],"down_context":[{"change":5,"pos":1694}],"flag":"BLOCK","start":1945,"end":1973,"inode":{"change":5,"pos":1640}}},"contents":[{"Add":"MakeError(NoTokens, Error)"},{"Add":""}]}},{"Replacement":{"inode":{"change":5,"pos":1640},"path":"src/hydra-queue-runner/token-server.hh","line":13,"ins":{"up_context":[{"change":5,"pos":1764}],"down_context":[{"change":5,"pos":1945}],"flag":"BLOCK","start":1974,"end":2184,"inode":{"change":5,"pos":1640}},"del":{"edges":[{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":1764},"to":{"change":5,"start":1764,"end":1945},"introduced_by":5}],"inode":{"change":5,"pos":1640}},"ins_contents":[{"Add":" available. Calling get(N) will return a Token object, representing"},{"Add":" ownership of N tokens. If the requested number of tokens is"},{"Add":" unavailable, get() will sleep until another thread returns a"},{"Add":" token. */"}],"del_contents":[{"Del":" available. Calling get() will return a Token object, representing\n ownership of a token. If no token is available, get() will sleep\n until another thread returns a token. */\n"}]}},{"Replacement":{"inode":{"change":5,"pos":1640},"path":"src/hydra-queue-runner/token-server.hh","line":20,"ins":{"up_context":[{"change":5,"pos":1966}],"down_context":[{"change":5,"pos":1994}],"flag":"BLOCK","start":2185,"end":2213,"inode":{"change":5,"pos":1640}},"del":{"edges":[{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":1966},"to":{"change":5,"start":1966,"end":1994},"introduced_by":5}],"inode":{"change":5,"pos":1640}},"ins_contents":[{"Add":" const size_t maxTokens;"}],"del_contents":[{"Del":" unsigned int maxTokens;\n"}]}},{"Replacement":{"inode":{"change":5,"pos":1640},"path":"src/hydra-queue-runner/token-server.hh","line":22,"ins":{"up_context":[{"change":5,"pos":1995}],"down_context":[{"change":2,"pos":2375}],"flag":"BLOCK","start":2214,"end":2241,"inode":{"change":5,"pos":1640}},"del":{"edges":[{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":1995},"to":{"change":5,"start":1995,"end":2032},"introduced_by":5}],"inode":{"change":5,"pos":1640}},"ins_contents":[{"Add":" Sync inUse{0};"}],"del_contents":[{"Del":" Sync curTokens{0};\n"}]}},{"Replacement":{"inode":{"change":5,"pos":1640},"path":"src/hydra-queue-runner/token-server.hh","line":26,"ins":{"up_context":[{"change":5,"pos":2081}],"down_context":[{"change":5,"pos":2148}],"flag":"BLOCK","start":2242,"end":2303,"inode":{"change":5,"pos":1640}},"del":{"edges":[{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":2081},"to":{"change":5,"start":2081,"end":2148},"introduced_by":5}],"inode":{"change":5,"pos":1640}},"ins_contents":[{"Add":" TokenServer(size_t maxTokens) : maxTokens(maxTokens) { }"}],"del_contents":[{"Del":" TokenServer(unsigned int maxTokens) : maxTokens(maxTokens) { }\n"}]}},{"Edit":{"inode":{"change":5,"pos":1640},"path":"src/hydra-queue-runner/token-server.hh","line":34,"atom":{"NewVertex":{"up_context":[{"change":5,"pos":2227}],"down_context":[{"change":5,"pos":2227}],"flag":"BLOCK","start":2304,"end":2328,"inode":{"change":5,"pos":1640}}},"contents":[{"Add":" size_t tokens;"},{"Add":""}]}},{"Replacement":{"inode":{"change":5,"pos":1640},"path":"src/hydra-queue-runner/token-server.hh","line":38,"ins":{"up_context":[{"change":5,"pos":2259}],"down_context":[{"change":5,"pos":2322}],"flag":"BLOCK","start":2329,"end":2435,"inode":{"change":5,"pos":1640}},"del":{"edges":[{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":2259},"to":{"change":5,"start":2259,"end":2322},"introduced_by":5}],"inode":{"change":5,"pos":1640}},"ins_contents":[{"Add":" Token(TokenServer * ts, size_t tokens, unsigned int timeout)"},{"Add":" : ts(ts), tokens(tokens)"}],"del_contents":[{"Del":" Token(TokenServer * ts, unsigned int timeout) : ts(ts)\n"}]}},{"Replacement":{"inode":{"change":5,"pos":1640},"path":"src/hydra-queue-runner/token-server.hh","line":41,"ins":{"up_context":[{"change":5,"pos":2332}],"down_context":[{"change":5,"pos":2430}],"flag":"BLOCK","start":2436,"end":2667,"inode":{"change":5,"pos":1640}},"del":{"edges":[{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":2332},"to":{"change":5,"start":2332,"end":2430},"introduced_by":5}],"inode":{"change":5,"pos":1640}},"ins_contents":[{"Add":" if (tokens >= ts->maxTokens)"},{"Add":" throw NoTokens(format(\"requesting more tokens (%d) than exist (%d)\") % tokens);"},{"Add":" auto inUse(ts->inUse.lock());"},{"Add":" while (*inUse + tokens > ts->maxTokens)"}],"del_contents":[{"Del":" auto curTokens(ts->curTokens.lock());\n while (*curTokens >= ts->maxTokens)\n"}]}},{"Replacement":{"inode":{"change":5,"pos":1640},"path":"src/hydra-queue-runner/token-server.hh","line":46,"ins":{"up_context":[{"change":5,"pos":2461}],"down_context":[{"change":5,"pos":2623}],"flag":"BLOCK","start":2668,"end":2832,"inode":{"change":5,"pos":1640}},"del":{"edges":[{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":2461},"to":{"change":5,"start":2461,"end":2623},"introduced_by":5}],"inode":{"change":5,"pos":1640}},"ins_contents":[{"Add":" if (!inUse.wait_for(ts->wakeup, std::chrono::seconds(timeout),"},{"Add":" [&]() { return *inUse + tokens <= ts->maxTokens; }))"}],"del_contents":[{"Del":" if (!curTokens.wait_for(ts->wakeup, std::chrono::seconds(timeout),\n [&]() { return *curTokens < ts->maxTokens; }))\n"}]}},{"Replacement":{"inode":{"change":5,"pos":1640},"path":"src/hydra-queue-runner/token-server.hh","line":50,"ins":{"up_context":[{"change":5,"pos":2678}],"down_context":[{"change":5,"pos":2754}],"flag":"BLOCK","start":2833,"end":2907,"inode":{"change":5,"pos":1640}},"del":{"edges":[{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":2678},"to":{"change":5,"start":2678,"end":2754},"introduced_by":5}],"inode":{"change":5,"pos":1640}},"ins_contents":[{"Add":" inUse.wait(ts->wakeup);"},{"Add":" *inUse += tokens;"}],"del_contents":[{"Del":" curTokens.wait(ts->wakeup);\n (*curTokens)++;\n"}]}},{"Replacement":{"inode":{"change":5,"pos":1640},"path":"src/hydra-queue-runner/token-server.hh","line":64,"ins":{"up_context":[{"change":5,"pos":2983}],"down_context":[{"change":5,"pos":3105}],"flag":"BLOCK","start":2908,"end":3030,"inode":{"change":5,"pos":1640}},"del":{"edges":[{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":2983},"to":{"change":5,"start":2983,"end":3105},"introduced_by":5}],"inode":{"change":5,"pos":1640}},"ins_contents":[{"Add":" auto inUse(ts->inUse.lock());"},{"Add":" assert(*inUse >= tokens);"},{"Add":" *inUse -= tokens;"}],"del_contents":[{"Del":" auto curTokens(ts->curTokens.lock());\n assert(*curTokens);\n (*curTokens)--;\n"}]}},{"Replacement":{"inode":{"change":5,"pos":1640},"path":"src/hydra-queue-runner/token-server.hh","line":74,"ins":{"up_context":[{"change":5,"pos":3223}],"down_context":[{"change":5,"pos":3263}],"flag":"BLOCK","start":3031,"end":3172,"inode":{"change":5,"pos":1640}},"del":{"edges":[{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":3223},"to":{"change":5,"start":3223,"end":3263},"introduced_by":5}],"inode":{"change":5,"pos":1640}},"ins_contents":[{"Add":" Token get(size_t tokens = 1, unsigned int timeout = 0)"},{"Add":" {"},{"Add":" return Token(this, tokens, timeout);"},{"Add":" }"},{"Add":""},{"Add":" size_t currentUse()"}],"del_contents":[{"Del":" Token get(unsigned int timeout = 0)\n"}]}},{"Replacement":{"inode":{"change":5,"pos":1640},"path":"src/hydra-queue-runner/token-server.hh","line":81,"ins":{"up_context":[{"change":5,"pos":3269}],"down_context":[{"change":5,"pos":3306}],"flag":"BLOCK","start":3173,"end":3232,"inode":{"change":5,"pos":1640}},"del":{"edges":[{"previous":"BLOCK","flag":"BLOCK | DELETED","from":{"change":5,"pos":3269},"to":{"change":5,"start":3269,"end":3306},"introduced_by":5}],"inode":{"change":5,"pos":1640}},"ins_contents":[{"Add":" auto inUse_(inUse.lock());"},{"Add":" return *inUse_;"}],"del_contents":[{"Del":" return Token(this, timeout);\n"}]}},{"Edit":{"inode":{"change":5,"pos":1640},"path":"src/hydra-queue-runner/token-server.hh","line":85,"atom":{"NewVertex":{"up_context":[{"change":5,"pos":3315}],"down_context":[],"flag":"BLOCK","start":3233,"end":3236,"inode":{"change":5,"pos":1640}}},"contents":[{"Add":""},{"Add":"}"}]}}],"owner":"grahamc","repo":"hydra2","token":"ZSL9+cgEsdaguhRECm8kzyI0JPslK9Jkka/D2TpCPSw=","login":null}

…

Dependencies

Change contents