pijul record should finish in seconds on 100MB input file, not minutes.

How to check your target:

$ cd
$ pijul init p1
$ cd p1
$ dd if=/dev/zero of=testfile bs=1024 count=102400
102400+0 records in
102400+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.668095 s, 157 MB/s
$ pijul add testfile
$ time pijul rec -am"."
Hash: 53POTSWQJ7ZFD4FMVYFIDXX7Q3JST5DYBFJUE5UUKCYHWA6CJLNQC

real    2m29.584s
user    2m28.754s
sys     0m0.160s
$

Modern cheap laptop on version 1.4.9:

$ cd
$ pijul init p1
$ cd p1
$ dd if=/dev/zero of=testfile bs=1024 count=102400
102400+0 records in
102400+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0,520491 s, 201 MB/s
$ pijul add testfile
$ time pijul rec -am"."
Hash: EJ4FUWN7SGEQBLSZ4HIDM5MXFNT7YDEHWKI5WEMS5A4GA5WBE3LAC

real	0m1,220s
user	0m0,995s
sys	0m0,209s
$ 

Latest 1.5.4 still affected.

Even 1.4.9 was affected in my tests, although I’m not fully certain that Nix and Pkg-config did the right thing there.

Anyway, I just released zstd-seekable 0.1.10 with the fix.

I am 100% sure 1.4.9 is ok

I gonna open issue back for history. I see the same slow result after patches.

Zstd-seekable 0.1.14 now embeds its own copy of ZStd and xxHash if it doesn’t find suitable versions using pkg-config.

These copies are compiled using the cc crate even on Windows, which also avoids embedding binary libs.

Will test another slow instance this evening and will close this discussion.

Fixed.

Finally! Compilation is also fixed on Windows. I submitted the issue to the ZStd folks, since we probably still want to get their updates in the future.

I am decided open this discussion again for history purpose. Current execution time on zstd 1.4.9+ is not optimal right now and pijul must be fast on any input and any zstd library version.

dd.sh script must end in second:

lambda:~/pijul-tests $ ./dd.sh 
+ source ./functions.sh
+ cd ..
+ rm -rf pijul-tests-data
+ mkdir pijul-tests-data
+ cd pijul-tests-data
+ pijul init repo
Repository created at repo
+ cd repo
+ dd if=/dev/zero of=a bs=1024 count=102400
102400+0 records in
102400+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0,306168 s, 342 MB/s
+ pijul add a
Tracked 1 path(s)
+ pijul record -am.
Hash: Y2IVDULQQ33LPAZDJMB5HKDZ7B4P2IAEP2VDEHV7VGSRFFYXJ6PQC

real	0m13,121s
user	0m12,935s
sys	0m0,075s
+ echo OK--dd
OK--dd
lambda:~/pijul-tests $ 

As a solution you could create a crate to collect all required information and do benchmarking on target system.

This is a blocker for stable release.

Last comment about the subject: any vcs should (must?) write a commit or record a patch in speed of hdd and ~10MB/s or so is not ok. I want to record any branch of linux kernel in seconds.

Zstd 1.5.5 released, fingers crossed.

zstd 1.5.5 is a fix. Issue closed.