The sound distributed version control system

#44 Use cargo2nix in flake

Closed on November 16, 2020
jwaksbaum on November 12, 2020

As suggested by @pmeunier here, I wrote a change to modify the flake.nix to use cargo2nix. This has a few advantages:

  • each dependent crate is built and therefore cached separately
  • we can add dependencies or patches to individual crates that we depend on, as is done here for zstd-seekable. This also allows us to omit some dependencies of crates we depend on that I assume are present in nixpkgs’s defaultCrateOverrides
  • Eventually there will be a mechanism for creating a devShell with transitive dependencies automatically present, but for now we need to specify them explicitly to get cargo build to succeed within the shell
jwaksbaum added a change on November 12, 2020
This change could not be retrieved at the moment
E725EC25DLBH46WJBDA3GZA4KUVB5KNWG6GUAE7JIIW3EYYKIMUAC
pmeunier on November 14, 2020

I’m extremely sorry, I love this change, but I couldn’t convert it to the new (1.0.0-alpha.1) change format. This new release breaks stuff, but is hopefully mathematically correct (whereas the previous one wasn’t). Would you mind:

  1. Cloning this repository again.
  2. Replacing flake.nix and flake.lock with your local versions.
  3. Recording and pushing to this discussion again.

Thanks a million!

jwaksbaum on November 14, 2020

Yup, happy to do that!

jwaksbaum on November 14, 2020

I’ve gotten this patch working again and building using the patches in #58, #59, and #60. Unfortunately, the pijul I build can’t push, giving me this backtrace:

name = 'pijul'
operating_system = 'unix:Unknown'
crate_version = '1.0.0-alpha.1'
explanation = '''
Panic occurred in file 'src/io/driver/mod.rs' at line 255
'''
cause = 'there is no reactor running, must be called from the context of Tokio runtime'
method = 'Panic'
backtrace = '''

   0: 0xaaaade56725c - core::option::expect_failed::he75a61737b600780
   1: 0xaaaade1fbe74 - tokio::io::poll_evented::PollEvented<E>::new::h8c4cd1b841fc6979
   2: 0xaaaade1fd418 - tokio::net::tcp::stream::TcpStream::new::h21e01152b1d8352c
   3: 0xaaaaddface60 - pijul::remote::ssh::Remote::connect::{{closure}}::h4c92cd5823a14eed
   4: 0xaaaaddf82958 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h759ec38d119c06d1
   5: 0xaaaaddf8647c - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hba415f6e368caeaf
   6: 0xaaaaddfa2e08 - pijul::commands::pushpull::Push::run::{{closure}}::hef976cfee4c398ed
   7: 0xaaaaddfb2050 - pijul::run::{{closure}}::h4174a6444cce861a
   8: 0xaaaaddf78d6c - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h01fc19c063e12fd0
   9: 0xaaaaddf00358 - tokio::runtime::enter::Enter::block_on::he821a3bbbbf5bb9c
  10: 0xaaaaddeb5ee4 - tokio::runtime::thread_pool::ThreadPool::block_on::hae664527612c58c3
  11: 0xaaaaddfcfbcc - tokio::runtime::context::enter::h60de3efb558fa6dd
  12: 0xaaaaddf0055c - tokio::runtime::handle::Handle::enter::h86d2e46e2b879ab3
  13: 0xaaaade06a87c - pijul::main::ha9754a027660ead0
  14: 0xaaaaddfffc58 - std::sys_common::backtrace::__rust_begin_short_backtrace::h3c89fd5ec7b78149
  15: 0xaaaade002e48 - std::rt::lang_start::{{closure}}::h18acb7d779155ef0
  16: 0xaaaade52b12c - std::rt::lang_start_internal::hb6930e9e2a2353c3
  17: 0xaaaade06a9a4 - main
  18: 0xffffabf62370 - __libc_start_main
  19: 0xaaaadde99238 - <unresolved>'''

My old pijul can’t push either. I’ll check-in soon to see if these problems are fixed so I can get this change pushed, or maybe try to fix them myselves.

ln on November 15, 2020

I wonder if https://github.com/nmattia/naersk would be a better fit?

pmeunier on November 15, 2020

This Tokio reactor problem is strange, I can’t reproduce. This is usually due to different versions of Tokio used at the same time, but I thought I had been careful not to do that.

One issue at the moment is that we’re waiting on a version of Hyper that works with Tokio 0.3.

jwaksbaum on November 15, 2020

Thanks for the tip on the tokio issue, I may have messed around with dependency versions on my own to fix the building issues instead of actually using the patches from the discussion I mentioned above, so it’s probably that.

jwaksbaum on November 15, 2020

@ln I don’t know tons about the differences between naersk and crate2nix, but I like the way that crate2nix creates a separate derivation for each dependent crate. I think it allows for more fine-grained caching and reuse, it allows you to specify dependences per-crate, and it avoids having to randomly update the cargo hash.

pmeunier on November 15, 2020

The difference is that Naersk lets Cargo drive the build, whereas Crate2nix relies on Nix only.

I wrote an ancestor to Crate2nix (called Carnix), and have used Naersk (or similar hacks), and I’m biased towards Crate2nix: although it doesn’t follow exactly what Cargo does, it is more Nixy, and I believe crates might be able to share more dependencies in the Nix store with Crate2nix.

jwaksbaum added a change on November 15, 2020
This change could not be retrieved at the moment
H462WA2SAOEOYKFHAYFYD4MVMACAB26O2A4ISBTAM3VMTKX3MCIAC
jwaksbaum added a change on November 15, 2020
This change could not be retrieved at the moment
HSENWI47VH726MGWNGASPKYPWKBVFBQDE522BU4B376ZOMH4MWWAC
jwaksbaum added a change on November 15, 2020
This change could not be retrieved at the moment
HIECBJLLEHLLZQ6KTKQGRDW7D3S7YOZGUQVXUABI4BHPFQITHHDQC
jwaksbaum on November 15, 2020

I’ve written two more changes, one that pins nixpkgs to release-20.09 instead of nixpkgs-unstable, and another that uses the mozilla overlay to be able to specify a Rust version independently from the version in nixpkgs. I’ve pinned it to the latest version, 1.47.0. I’ll also be putting the built artifacts on my cachix.

I should mention that I only have an aarch64 NixOS system and a x86_64 non-NixOS Nix system, no darwin system, so I haven’t been able to test anything on darwin.

I think a great next step for build-system/CI/Nix stuff would be setting up a pijul cachix, and maybe using nixbuild.net for CI.

jwaksbaum added a change on November 15, 2020
This change could not be retrieved at the moment
Z52GM2QYDX3RXJ345GA4A6WKQSMNBAJYYRCEV4KZQNVXVKVUOLOQC
jwaksbaum added a change on November 15, 2020
This change could not be retrieved at the moment
7Y7P3EAIIOPHX2SMF74K3M4S32Z2MTZVKZN7HZOZ74PXGBDF5YJAC
pmeunier on November 15, 2020

Done, thanks! I just looked up nixbuild.net, it seems nice. One issue, as always, is getting a specific version from Pijul, but once we’re more stable, this would be really cool.

pmeunier closed this discussion on November 15, 2020
cole-h on November 15, 2020

It seems like the Update Cargo.lock change succumbed to one of the issues discussed in #51 – on a fresh clone, I get conflict markers. I don’t know if you want to unrecord that change and then record a cargo update yourself, @pmeunier.

cole-h on November 15, 2020

Seeing #72, it seems like some/most of the changes from this discussion have issues.

(Idea about why the conflict markers don’t show on the nest: maybe parsed as HTML tags, just like <nixpkgs> once was?)

pmeunier on November 16, 2020

Alright I really need to implement conflict checks on the Nest. This seems a top priority.

I’m reopening this until the conflicts are resolved.

pmeunier reopened this discussion on November 16, 2020
cole-h added a change on November 16, 2020
This change could not be retrieved at the moment
6WMAVF4YCJJXKFLUMUCNPSTUHD4J2EXSMSYGKPIFJC4NIX3BZYRAC
cole-h added a change on November 16, 2020
G734WNM64AR5BLAZMN5MDPKSFTYXTUQR6MAGB32NRBC5FXFRWSJAC
main
cole-h on November 16, 2020

I took the liberty of recreating the change that switches flake.nix to using crate2nix (with some opinionated style choices). Thanks for doing most of the legwork, Jake!

(The previous, removed-from-discussion change included version bumps in the Cargo.toml of pijul and libpijul – didn’t belong in that change.)

pmeunier on November 16, 2020

Again, thanks a lot for your patience and your changes!

Sorry about the bumps, the recent change in format was pretty massive and impacted most parts of libpijul.

pmeunier closed this discussion on November 16, 2020
cole-h on November 16, 2020

@pmeunier #C4I5JID2TIJMLIYM3KZANHFWVITZF3EGVTQFN5C5TAZAO5SI3E6QC introduced another conflict. Since you pushed my above change, there should already be a compatible shell.nix, as well as no more version or cargoSha256 in the flake.nix.

pmeunier on November 16, 2020

Thanks, I had made that change before looking at the repository and approving your change.

At the moment I’m trying to get Thrussh back together, with multiple channels for the two version of Tokio.

jwaksbaum on November 16, 2020

I’m not sure exactly what happened here, did I make a mistake in the way I recorded and unrecorded? I’d like to understand if I did so I can not make it again the next time. Thanks for fixing it Cole! Let me know if there’s anything else I can do here, I still have a version locally that worked for me at least.

jwaksbaum on November 16, 2020

I also see that my change that pins the rust version with the mozilla overlay got lost, or maybe we decided we didn’t want to use it? Fine either way, but as I still have it locally I can repush it if there was another reboot of the history as there appears to have been

cole-h on November 16, 2020

I don’t think you did anything wrong – it’s just that you used a pijul version affected by an earlier issue with unrec causing an invalid change to be generated (I think). That’s why history was rebooted again: it was mucked with due to some issue that was fixed but still present in the history.

I purposefully didn’t use the Mozilla overlay because I don’t think there’s much point. 20.09 currently has Rust 1.45.2, which is recent enough I would think. If we wanted a more recent version of Rust (e.g. 1.47.0), wouldn’t we want to use the nix{os,pkgs}-unstable channel for our packages, instead of the overlay? Mostly just a matter of principle, but I also don’t like relying on any more non-flake inputs than necessary (crate2nix was definitely necessary, so we don’t have to update the cargoSha256 all the time) :P

jwaksbaum on November 16, 2020

Oh I understand – thanks for clarifying what happened!

About the overlay, I think it’s nice to be able to easily change the version of Rust you’re using without changing the versions of all our other dependencies we get from Nixpkgs. I don’t think it’s an issue at all to fetch from more sources, regardless of whether or not they are packaged as flakes. I think part of the point of flakes is to decentralize the ecosystem a bit, and I think it’s good to fetch our Rust “from the source” from Mozilla, and to decouple it from Nixpkgs. We could imagine, if we wanted to, running a build on our CI against multiple rust versions to ensure we support them, or against Nightly to make sure upcoming versions won’t break things, all of which is a lot easier if you can swap your Rust version independently from Nixpkgs.

I also had two other things I wanted to discuss about the current flake.nix:

  • pijul has a direct dependency on openssl and libsodium, but I think those aren’t necessary since they are dependencies of one of pijul’s dependencies, not dependencies of pijul directly.
  • crate2nix is built using whatever version of nixpkgs we are using. I think that this will force us to rebuild crate2nix from scratch, whereas if we use the version of nixpkgs they have pinned we will be able to use the binaries they store in their cachix. Though I’m on aarch64, so I haven’t confirmed that this actually works, it feels like the right thing to do to not override their pinned Nixpkgs.
cole-h on November 16, 2020

CI is something I hadn’t thought about. I’ll open a new discussion addressing this and the below points.

  • Indeed, those dependencies aren’t necessary. At one point, the build failed without openssl (and so I added libsodium to be safe), but that doesn’t seem to be the case any longer.
  • A fair point, but this requires one to add the crate2nix cachix and its key to list of substituters (either temporarily by the command line flag, or permanently with a nix.conf). This will result in more bandwidth and space being used, however. Using their niv-pinned nixpkgs and starting up nix develop, I was required to download 320MiB more (which extracted to 1205MiB). I personally feel that space and bandwidth are much more costly than having to build crate2nix (which took me 2m31s when running cargo build -j1) every time we bump our nixpkgs flake input (which shouldn’t be very often).

Let’s continue this in #87 :)