E725EC25DLBH46WJBDA3GZA4KUVB5KNWG6GUAE7JIIW3EYYKIMUAC
I’m extremely sorry, I love this change, but I couldn’t convert it to the new (1.0.0-alpha.1) change format. This new release breaks stuff, but is hopefully mathematically correct (whereas the previous one wasn’t). Would you mind:
flake.nix
and flake.lock
with your local versions.Thanks a million!
Yup, happy to do that!
I’ve gotten this patch working again and building using the patches in #58, #59, and #60. Unfortunately, the pijul I build can’t push, giving me this backtrace:
name = 'pijul'
operating_system = 'unix:Unknown'
crate_version = '1.0.0-alpha.1'
explanation = '''
Panic occurred in file 'src/io/driver/mod.rs' at line 255
'''
cause = 'there is no reactor running, must be called from the context of Tokio runtime'
method = 'Panic'
backtrace = '''
0: 0xaaaade56725c - core::option::expect_failed::he75a61737b600780
1: 0xaaaade1fbe74 - tokio::io::poll_evented::PollEvented<E>::new::h8c4cd1b841fc6979
2: 0xaaaade1fd418 - tokio::net::tcp::stream::TcpStream::new::h21e01152b1d8352c
3: 0xaaaaddface60 - pijul::remote::ssh::Remote::connect::{{closure}}::h4c92cd5823a14eed
4: 0xaaaaddf82958 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h759ec38d119c06d1
5: 0xaaaaddf8647c - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hba415f6e368caeaf
6: 0xaaaaddfa2e08 - pijul::commands::pushpull::Push::run::{{closure}}::hef976cfee4c398ed
7: 0xaaaaddfb2050 - pijul::run::{{closure}}::h4174a6444cce861a
8: 0xaaaaddf78d6c - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h01fc19c063e12fd0
9: 0xaaaaddf00358 - tokio::runtime::enter::Enter::block_on::he821a3bbbbf5bb9c
10: 0xaaaaddeb5ee4 - tokio::runtime::thread_pool::ThreadPool::block_on::hae664527612c58c3
11: 0xaaaaddfcfbcc - tokio::runtime::context::enter::h60de3efb558fa6dd
12: 0xaaaaddf0055c - tokio::runtime::handle::Handle::enter::h86d2e46e2b879ab3
13: 0xaaaade06a87c - pijul::main::ha9754a027660ead0
14: 0xaaaaddfffc58 - std::sys_common::backtrace::__rust_begin_short_backtrace::h3c89fd5ec7b78149
15: 0xaaaade002e48 - std::rt::lang_start::{{closure}}::h18acb7d779155ef0
16: 0xaaaade52b12c - std::rt::lang_start_internal::hb6930e9e2a2353c3
17: 0xaaaade06a9a4 - main
18: 0xffffabf62370 - __libc_start_main
19: 0xaaaadde99238 - <unresolved>'''
My old pijul can’t push either. I’ll check-in soon to see if these problems are fixed so I can get this change pushed, or maybe try to fix them myselves.
I wonder if https://github.com/nmattia/naersk would be a better fit?
This Tokio reactor problem is strange, I can’t reproduce. This is usually due to different versions of Tokio used at the same time, but I thought I had been careful not to do that.
One issue at the moment is that we’re waiting on a version of Hyper that works with Tokio 0.3.
Thanks for the tip on the tokio issue, I may have messed around with dependency versions on my own to fix the building issues instead of actually using the patches from the discussion I mentioned above, so it’s probably that.
@ln I don’t know tons about the differences between naersk
and crate2nix
, but I like the way that crate2nix
creates a separate derivation for each dependent crate. I think it allows for more fine-grained caching and reuse, it allows you to specify dependences per-crate, and it avoids having to randomly update the cargo hash.
The difference is that Naersk lets Cargo drive the build, whereas Crate2nix relies on Nix only.
I wrote an ancestor to Crate2nix (called Carnix), and have used Naersk (or similar hacks), and I’m biased towards Crate2nix: although it doesn’t follow exactly what Cargo does, it is more Nixy, and I believe crates might be able to share more dependencies in the Nix store with Crate2nix.
H462WA2SAOEOYKFHAYFYD4MVMACAB26O2A4ISBTAM3VMTKX3MCIAC
HSENWI47VH726MGWNGASPKYPWKBVFBQDE522BU4B376ZOMH4MWWAC
HIECBJLLEHLLZQ6KTKQGRDW7D3S7YOZGUQVXUABI4BHPFQITHHDQC
I’ve written two more changes, one that pins nixpkgs
to release-20.09
instead of nixpkgs-unstable
, and another that uses the mozilla overlay to be able to specify a Rust version independently from the version in nixpkgs
. I’ve pinned it to the latest version, 1.47.0
. I’ll also be putting the built artifacts on my cachix.
I should mention that I only have an aarch64
NixOS system and a x86_64
non-NixOS Nix system, no darwin system, so I haven’t been able to test anything on darwin.
I think a great next step for build-system/CI/Nix stuff would be setting up a pijul cachix, and maybe using nixbuild.net for CI.
Z52GM2QYDX3RXJ345GA4A6WKQSMNBAJYYRCEV4KZQNVXVKVUOLOQC
7Y7P3EAIIOPHX2SMF74K3M4S32Z2MTZVKZN7HZOZ74PXGBDF5YJAC
Done, thanks! I just looked up nixbuild.net, it seems nice. One issue, as always, is getting a specific version from Pijul, but once we’re more stable, this would be really cool.
Alright I really need to implement conflict checks on the Nest. This seems a top priority.
I’m reopening this until the conflicts are resolved.
6WMAVF4YCJJXKFLUMUCNPSTUHD4J2EXSMSYGKPIFJC4NIX3BZYRAC
G734WNM64AR5BLAZMN5MDPKSFTYXTUQR6MAGB32NRBC5FXFRWSJAC
I took the liberty of recreating the change that switches flake.nix to using crate2nix (with some opinionated style choices). Thanks for doing most of the legwork, Jake!
(The previous, removed-from-discussion change included version bumps in the Cargo.toml of pijul and libpijul – didn’t belong in that change.)
Again, thanks a lot for your patience and your changes!
Sorry about the bumps, the recent change in format was pretty massive and impacted most parts of libpijul.
@pmeunier #C4I5JID2TIJMLIYM3KZANHFWVITZF3EGVTQFN5C5TAZAO5SI3E6QC introduced another conflict. Since you pushed my above change, there should already be a compatible shell.nix
, as well as no more version
or cargoSha256
in the flake.nix
.
Thanks, I had made that change before looking at the repository and approving your change.
At the moment I’m trying to get Thrussh back together, with multiple channels for the two version of Tokio.
I’m not sure exactly what happened here, did I make a mistake in the way I recorded and unrecorded? I’d like to understand if I did so I can not make it again the next time. Thanks for fixing it Cole! Let me know if there’s anything else I can do here, I still have a version locally that worked for me at least.
I also see that my change that pins the rust version with the mozilla overlay got lost, or maybe we decided we didn’t want to use it? Fine either way, but as I still have it locally I can repush it if there was another reboot of the history as there appears to have been
I don’t think you did anything wrong – it’s just that you used a pijul version affected by an earlier issue with unrec
causing an invalid change to be generated (I think). That’s why history was rebooted again: it was mucked with due to some issue that was fixed but still present in the history.
I purposefully didn’t use the Mozilla overlay because I don’t think there’s much point. 20.09 currently has Rust 1.45.2, which is recent enough I would think. If we wanted a more recent version of Rust (e.g. 1.47.0), wouldn’t we want to use the nix{os,pkgs}-unstable
channel for our packages, instead of the overlay? Mostly just a matter of principle, but I also don’t like relying on any more non-flake inputs than necessary (crate2nix was definitely necessary, so we don’t have to update the cargoSha256
all the time) :P
Oh I understand – thanks for clarifying what happened!
About the overlay, I think it’s nice to be able to easily change the version of Rust you’re using without changing the versions of all our other dependencies we get from Nixpkgs. I don’t think it’s an issue at all to fetch from more sources, regardless of whether or not they are packaged as flakes. I think part of the point of flakes is to decentralize the ecosystem a bit, and I think it’s good to fetch our Rust “from the source” from Mozilla, and to decouple it from Nixpkgs. We could imagine, if we wanted to, running a build on our CI against multiple rust versions to ensure we support them, or against Nightly to make sure upcoming versions won’t break things, all of which is a lot easier if you can swap your Rust version independently from Nixpkgs.
I also had two other things I wanted to discuss about the current flake.nix
:
pijul
has a direct dependency on openssl
and libsodium
, but I think those aren’t necessary since they are dependencies of one of pijul
’s dependencies, not dependencies of pijul
directly.crate2nix
is built using whatever version of nixpkgs
we are using. I think that this will force us to rebuild crate2nix
from scratch, whereas if we use the version of nixpkgs
they have pinned we will be able to use the binaries they store in their cachix. Though I’m on aarch64
, so I haven’t confirmed that this actually works, it feels like the right thing to do to not override their pinned Nixpkgs.CI is something I hadn’t thought about. I’ll open a new discussion addressing this and the below points.
openssl
(and so I added libsodium
to be safe), but that doesn’t seem to be the case any longer.crate2nix
cachix and its key to list of substituters (either temporarily by the command line flag, or permanently with a nix.conf). This will result in more bandwidth and space being used, however. Using their niv
-pinned nixpkgs
and starting up nix develop
, I was required to download 320MiB more (which extracted to 1205MiB). I personally feel that space and bandwidth are much more costly than having to build crate2nix (which took me 2m31s when running cargo build -j1
) every time we bump our nixpkgs flake input (which shouldn’t be very often).Let’s continue this in #87 :)
As suggested by @pmeunier here, I wrote a change to modify the
flake.nix
to usecargo2nix
. This has a few advantages:zstd-seekable
. This also allows us to omit some dependencies of crates we depend on that I assume are present innixpkgs
’sdefaultCrateOverrides
devShell
with transitive dependencies automatically present, but for now we need to specify them explicitly to getcargo build
to succeed within the shell