Moving the discussion from #44.

So far, this includes two changes: one that uses the Mozilla overlay (thanks, @jwaksbaum), and one that removes two unnecessary buildInputs.

We’ll also be discussing the merits of using crate2nix’s pinned nixpkgs (allowing people to use the crate2nix cachix for a precompiled binary) rather than using our flake input (and forcing users to build crate2nix themselves at least once).

Responding to what you said about crate2nix:

A fair point, but this requires one to add the crate2nix cachix and its key to list of substituters (either temporarily by the command line flag, or permanently with a nix.conf). This will result in more bandwidth and space being used, however. Using their niv-pinned nixpkgs and starting up nix develop, I was required to download 320MiB more (which extracted to 1205MiB). I personally feel that space and bandwidth are much more costly than having to build crate2nix (which took me 2m31s when running cargo build -j1) every time we bump our nixpkgs flake input (which shouldn’t be very often). I think these are both totally fair points, especially the huge amount of space and bandwidth versus the short compilation time.

Reading this did make me wonder though: why the heck is it downloading over a gigabyte for crate2nix when it should be a static binary? It turns out the crate2nix default.nix is bringing along all of cargo, which includes llvm, etc. So maybe that’s part of why you said you don’t like using non-flake inputs :). Manually editing it brings the runtime dependencies to just the binary, glibc, and 2 libs, weighing in at 35M, compared to 1.3G.

/nix/store/62iaip0raq8gr8vd1a54z63hgmkrliwy-libunistring-0.9.10
/nix/store/0ryslbjqqjfnhy9g8i3cnkfn7pn6svpk-libidn2-2.3.0
/nix/store/1ygglqib3lg0wywg95rk2kmxvl97zp08-glibc-2.30
/nix/store/1hgq5amcy5bmqnvxchb91znxgca883pg-rust_crate2nix-0.8.0

It seems like a big gotcha for them to bundle all that by default, so maybe we want to open an issue upstream. But at any rate, if we have a way to work around it such that it wasn’t so much space and bandwidth, would that change your opinion on using their vs our nixpkgs?

That being said, this would all be kind of moot unless you’re configured to use their cachix. But still, you would just need to compile it once, and then all the build-time dependencies could be garbage-collected. And it would stay the same even if we upgraded Nixpkgs, changing only if we upgraded crate2nix specifically.

In general, I would say that we should treat crate2nix like a developer dependency, and it feels strange that if, for example, we bumped our Nixpkgs version we would recompile a tool we depend on just to generate files. It feels like the right thing to do to use the “released binary” instead of “recompiling from source”, so if there isn’t an overriding reason against it (like needing to download and store 1GB), that would be my preference.

This concern might be totally eliminated by getting crate2nix into Nixpkgs. According to https://github.com/kolloch/crate2nix/issues/137, there isn’t really anything blocking this aside from somebody doing it. The “downside” would be that it would necessitate us switching to unstable. (Not really, though; there have been “package inits” merged into release branches in the past.)

I’ll get crate2nix packaged (should be fairly straightforward), seeing as there hasn’t been a whole lot of movement recently.

If you were able to shrink the closure, that would minimize my complaints, but having it in Nixpkgs itself would eliminate them altogether (and also allow us to drop a non-flake input ;) ).

Aside for Pierre when he stumbles upon this thread: I think Nest might have a “duplicate removal” issue – maybe it’s running on an older version of Pijul or something? If you check #H5ETV2RC4LQT6ZYE7KVZHEXASMVZDFCPGWNGXET7DZDOOEFUJ47QC above, it has rustPackages.cargo and friends removed twice.

Sorry for the spurious changes in this discussion, still getting the hang of this.

HGQ3YDCW53XDCYMO7G22ZK4AFGSJKVEZSO32CJKDCO376HNLBXJAC is my attempt to fix the crate2nix closure situation.

In 4G2JOCKDMW37FVVKE44KIP63VCYYKIBXLAH6GU7GF67P6BXC6HEAC I modified your change to use the mozilla version of rust to build pijul itself; the way it is I think it’s only used in the devShell. To do that I used overlays again, which I noticed you took out, but that’s just because I couldn’t think of another way. Feel free to change that of course.

I realized I removed rls etc from the devShell, I’ll fix that in a second.

No worries, as you can see, I’ve done the same many times… (It’s a little embarrassing that “remove from discussion” doesn’t remove it altogether, and just hides it from pijul pull haha.)

Good catch. The reason I didn’t rely on overlays is because… I don’t like overlays. This stems from the pre-flakes days where I would sometimes shoot myself in the foot by having ~/.config/nixpkgs/overlays/*.nix and it occasionally masking issues. Unfortunately, there isn’t a(n easy) way to change rustc and cargo to point to the version provided by rustChannel except for an overlay.

re: rls – rustChannel.rust provides rls, rust-analyzer, etc.

Here’s another thought, not sure if it belongs here or a separate discussion: it’d be nice if we hid the nix-specific things from the rust build by filtering src. That way changing flake.nix wouldn’t cause an unnecessary rebuild.

Thanks for your help. I’m pushing changes one last time to supercede yours (just minor style fixups, since I love to nitpick).

re filter src – There’s no src attribute any more, so I don’t think there’s anything to filter?

EDIT: In the end, the changes to apply are:

VN6L65VRWLKTIXY7XD7OOZBMNKNSIEJG6PJUX5NKKYVYGVG4DFTAC 2GNO2PLCZ3BM5RRRSPLGVWEWHOOTVT4VKFBNNQMUKOKF3VXL3ZFQC ZWPPIJXFKEL3RP2MEYGB3BPTE73GYRW6FSRZDQKFBJFOK4JBNQ4AC

Thanks for your help. I’m pushing changes one last time to supercede yours (just minor style fixups, since I love to nitpick).

Looks good to me, thanks for cleaning it all up!

re filter src – There’s no src attribute any more, so I don’t think there’s anything to filter?

I think Cargo.nix still has src attributes for each crate, so you can override that, for example as is done in crate2nix. In terms of what we would filter, we could filter the src we are overriding, but we could also filter self, which as a path is the source of the flake itself. That might be better because the flake input filters some stuff out already: right now it uses git if the flake is a git repository, so if it supported pijul that would filter out ignored things.

I think actually all of libsodium openssl etc. from before are necessary for building within a devShell.

Also, I’m seeing conflicts between the three changes here, but I’m not sure why…

I see what you’re talking about. However, none of those src attributes reference the top-level repo (e.g. containing pijul, libpijul, and pijul-macros) – they all deal with the subfolders, in which there shouldn’t be anything aside from Rust-related files. Maybe I’m missing something.

(Above re-push is because we still need openssl in the buildInputs of the devShell – I built fine inside a nix develop -i shell without libsodium.)

re conflicts – I think this might be on your end? A fresh clone and pijul pull .... doesn’t display any conflict markers on my end.

Thanks!

@jwaksbaum: are you sure you don’t have former versions of these changes? I don’t see any conflict when I pull from this discussion.

I’m having a problem with the flake.nix, but that might be because I’m not yet too familiar with flakes.

Why do nix shell and nix develop insist on compiling the entire Pijul before giving me a shell? Is that intended, or is there a way to not do that, and just give me a shell with the buildInputs in scope?

I just did a fresh clone, so conflicts are gone.

As for filtering the source, I think you’re right that crate2nix is smarter than I thought and uses the subfolders. I feel like I’m constantly rebuilding libpijul for no good reason, but I’ll have to figure out the real cause if it happens again.

@pmeunier nix develop shouldn’t compile pijul, it should drop you into a shell with the dependencies. nix shell in the other handle will drop you into a shell with pijul. This is notably different from nix-shell which makes it all very confusing. Are you seeing it compile pijul for nix develop? I’m not sure if it will compile all the ddependent crates for that, I’ll have to check.

It shouldn’t be compiling for nix develop. I can’t reproduce it here, either. Is there any chance you’re just seeing “something Rust-y” compile? Because crate2nix does need to be compiled (it’s not just a shell script – it’s written in Rust!) “once”.

Alright, it might be this, then. I saw nix shell compile Pijul, tried nix develop and it was still compiling stuff. Thanks for the hints.

(These were all applied, so closing.)

FWIW: https://github.com/NixOS/nixpkgs/pull/104027 was merged into Nixpkgs earlier today, and https://github.com/NixOS/nixpkgs/pull/104106 is the backport to 20.09, which should be merged Soon^TM.

Once it gets merged, we’ll be able to drop the custom derivation and nobody should have to build crate2nix on their own anymore.