Reimplementation of Pijul in C, for education, fun and absolutely no profit

#10 Think of ways to reduce memory usage

Opened by laumann on July 21, 2023
laumann on July 21, 2023
valgrind --leak-check=full build/ani change YM2LC6QP2D7TK3JUKOXWUWTQEGMCEGQDGPTOZPWC3BRDFNRKSZWQC

reports (at least):

==15877==   total heap usage: 326 allocs, 326 frees, 50,066,715 bytes allocated

ideas for reducing this usage:

  • don’t read in contents of dependencies, unless needed
  • don’t deserialize hunks in dependencies (never needed for the change subcommand)
laumann on July 23, 2023

Idea 1: In change we could defer deserializing a change’s hunks before it’s known to be needed. For the #YM2LC6QP2D7TK3JUKOXWUWTQEGMCEGQDGPTOZPWC3BRDFNRKSZWQC change, all hunks in all its dependencies are deserialized and all take up thousands (in the span of 8-11 Kbytes) memory, and is never used.

EDIT: Some data:

$ make check2 |grep basehunk
../hunk.c:50: xmalloc(7128) for 'sizeof(struct basehunk) * len' 
../hunk.c:50: xmalloc(9720) for 'sizeof(struct basehunk) * len' 
../hunk.c:50: xmalloc(8424) for 'sizeof(struct basehunk) * len' 
../hunk.c:50: xmalloc(11664) for 'sizeof(struct basehunk) * len' 
../hunk.c:50: xmalloc(9720) for 'sizeof(struct basehunk) * len' 
laumann on July 23, 2023

Idea 2: Some kind of string interning, or at least encoding interning - for the same change (#YM2LC6QP2D7TK3JUKOXWUWTQEGMCEGQDGPTOZPWC3BRDFNRKSZWQC) the string UTF-8 is decoded and malloc()’ed 72 times.

EDIT: A similar thing could maybe be done with the path component of struct local - the same change has 50 string allocations for the string change.c.

laumann on July 23, 2023

Idea 3 (that really should be the first idea): same vein as idea 1, avoid decompressing change contents until absolutely necessary, this is probably where the biggest gains are to be made….