grahamc/hydra2 - Change 3D3F5I744YILPYZYZQHGADFNYIM7RESDLSCDULH5WVIIPTYD5GYQC

Fix race between hydra-eval-jobs and hydra-update-gc-roots

If hydra-eval-jobs creates a new root, and hydra-update-gc-roots runs before hydra-evaluator has had a chance to add the corresponding build to the database, then hydra-update-gc-roots will remove the root. If subsequently the Nix garbage collector kicks in, it may remove the build's .drv file before the build is performed. Since evaluation of the Nixpkgs and NixOS jobsets nowadays takes a lot of time (e.g. an hour), the probability of this happening is fairly high.

The quick fix is not to delete roots that are less than a day old. So long as evaluation doesn't take longer than a day, this should be fine ;-)

Fixes #166.

Created by Eelco Dolstra on July 14, 2014

3D3F5I744YILPYZYZQHGADFNYIM7RESDLSCDULH5WVIIPTYD5GYQC

Dependencies

In channels

main

Change contents

Insertion in src/script/hydra-update-gc-roots at line 5 [3.472]
[3.544]
[3.544]
```
use File::stat;
```

Replacement in src/script/hydra-update-gc-roots at line 52 [3.472]

B:BD[3.1148] → [4.0:149]

# Read the current GC roots.  We need to do that here so that we don't
# delete roots that were added while we were determining the desired
# roots.

[3.1148]

[4.149]

# Read the current GC roots.

Insertion in src/script/hydra-update-gc-roots at line 124 [3.472]
[5.2766]
[3.2064]
```
my $now = time();
```

Replacement in src/script/hydra-update-gc-roots at line 130 [3.472]

B:BD[3.2221] → [3.2221:2267]

B:BD[3.2267] → [2.19656:19681]

∅:D[2.19681] → [6.0:78]

B:BD[5.2864] → [6.0:78]

        print STDERR "removing root $path\n";
        $rootsDeleted++;
        unlink "$gcRootsDir/$link" or warn "cannot remove $gcRootsDir/$link";

[3.2221]

[5.2942]

        # Don't delete roots that are less than a day old, to prevent
        # a race where hydra-eval-jobs has added a root but
        # hydra-evaluator hasn't added them to the database yet.
        if (lstat($path)->ctime < $now - 24 * 60 * 60) {
            print STDERR "removing root $path\n";
            $rootsDeleted++;
            unlink "$gcRootsDir/$link" or warn "cannot remove $gcRootsDir/$link";
        } else {
            print STDERR "NOT removing recent root $path\n";
            $rootsKept++;
        }