The sound distributed version control system

#199 [offtopic] high mem usage on "pijul log"

Closed on December 11, 2020
tankf33der on December 8, 2020

I have created repo for linux 5.x series kernel. And successfully created three records from incremental patches:

5.0 -> 5.0.1 -> 5.0.2

Repo dir is 1.6GB in sum for 67K files.

One small (?) issue is pijul log eats 1.3GB of RAM to show three records, rest command’s memory appetites are OK in general. For example:

  • pijul rec -am"." on not (!) modified repo eats only 200MB;
  • pijul diff --short on modified repo eats only 200MB;
  • pijul rec -am"5.0.2" after applied patch to file tree eats only 200MB;
  • pijul add -r * eats nothing after applied patch and longest run operation;
pmeunier on December 8, 2020

Thanks! This is because it loads up the entire patch in memory. That’s quite naive and could be easily fixed.

tankf33der on December 8, 2020

Could be pijul add -r * made be run faster?

tankf33der on December 8, 2020

Attention, you broken db access right now:

$ pijul log
# press q to exit editor
$ pijul log --hash-only
Error: Pristine locked
$
pmeunier on December 11, 2020

I wasn’t patient enough to wait for pijul add -r to complete on the linux kernel sources, so I fixed it. Now I don’t have the time to hit ctrl+c.

pmeunier added a change on December 11, 2020
OJZWJUF2TCGZ7RFVY6FPKBS5P3C4BGHZDPVH775OHVNVFMJICKNQC
main
pmeunier on December 11, 2020

Oh, and I just tested, pijul log with the first initial giant patch takes 9Mb now. Thanks a lot for this test! It was super helpful.

pmeunier closed this discussion on December 11, 2020
tankf33der on December 11, 2020

Testing.

Now when applied new patch to linux kernel tree cant to add:

$ pi add -r *
Error: File already in repository: arch

Before it silently run thru all tree without fatal break.

tankf33der on December 16, 2020

Fixed.

tankf33der on December 16, 2020

Will you make performance boost for other commands too?

  • pijul diff
  • pijul record -am“patch 5.0.7 applied“

Lets skip for now slow pull and push commands on nest and count it cant handle such huge traffic.

pmeunier on December 17, 2020

I’d love to. One of the things I could do is to use the modification times, but that’s quite tricky.

tankf33der on December 17, 2020

Just do pijul diff in multi core way how you did for add and rest add to your road map for next year.