#878 Push via SSH: failed to fill whole buffer
Opened by descendingnerdliness on February 9, 2024
descendingnerdliness on February 9, 2024
BTW: I am using pijul 1.0.0-beta.8
tankf33der on February 10, 2024
How have you created such a huge repo?
descendingnerdliness on February 10, 2024
It is a logsec research database with notes including many papers in pdf. There are over 700 text files of notes and about 250 pdfs of 10 to 30 pages, plus the logsec internals that link it all.
tankf33der on February 10, 2024
How much is .pristine dir inside?
descendingnerdliness on February 11, 2024
Only 3MB. I just started trying pijul. Logsec has git integrated, so it is unavoidable.
descendingnerdliness on February 20, 2024
Hey, Anything I can help with on this?
I am currently stuck with git. Can’t seem to get fossil to work with my existing repos.
descendingnerdliness closed this discussion on February 20, 2024
descendingnerdliness reopened this discussion on February 20, 2024
tankf33der on February 20, 2024
How this is possible 600MB repo and so tiny .pijul dir?!!?
tankf33der on February 20, 2024
BTW pijul push -a name@server:/extra/phd i am not sure this is correct call.
tankf33der on February 20, 2024
is it ssh or https ?
tankf33der on February 20, 2024
show me pijul remote output
tankf33der on February 20, 2024
I have repeated error, this is good.
$ pijul rec -am. && pijul push -a
Hash: RSS3YEVMYBJRF7HHVFNTV7LB55Y6XI5HWWVO5TKJUN7HEW5OHJKQC
Uploading changes [==================================================] 1/1 [00:00:00] Error: failed to fill whole buffer
Error: Remote exited with status 1
$
tankf33der on February 20, 2024
my parameters:
$ du -sh r1
1,6G r1
$ cd r1
$ du -sh .pijul
491M .pijul
tankf33der on February 20, 2024
change subject of this discussion: [REPEATED] Push via SSH: failed to fill whole buffer
tankf33der on February 20, 2024
250MB change the same error.
descendingnerdliness on February 20, 2024
I am sorry, but I don’t understand what you want. Please tell me the commands to run.
tankf33der on February 20, 2024
# clone over ssh
$ cp /usr/bin/[a-g]* .
$ pijul add -r .
$ pijul rec -am.
Hash: GXU5LCWP73OKIS3BKPIJC5I5GSESW3DASX5FU2BO2WCIZ36CWCDQC
$ pijul push -a
ERROR. BOOM.
tankf33der on February 20, 2024
now i am cutting size of repo to find minimal size when error occurs.
tankf33der on February 20, 2024
11MB file failed too.
tankf33der on February 20, 2024
DEBUG OUTPUT
$ RUST_LOG="pijul=debug,libpijul=debug,sanakirja=debug" pijul push -a
[2024-02-20T17:38:12Z DEBUG pijul_repository] "/home/mpech/r1/.pijul"
[2024-02-20T17:38:12Z DEBUG pijul_repository] max_files = 64
[2024-02-20T17:38:12Z DEBUG pijul::commands::pushpull] Config { default_remote: Some("mpech@127.0.0.1:repo1"), extra_dependencies: [], remotes: [], hooks: Hooks { record: [] }, unrecord_changes: None, reset_overwrites_changes: None, colors: None, pager: None }
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] BEGIN_TXN root = 1
[2024-02-20T17:38:12Z DEBUG sanakirja::environment] CHECKING CRC 1077489304 1077489304
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] n_roots = 2
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] initial free_page 9000
[2024-02-20T17:38:12Z DEBUG pijul::commands::pushpull] remote_channel = "main" None
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] ssh_remote: Captures({0: 0..21/"mpech@127.0.0.1:repo1", 1: None, 2: 0..6/"mpech@", 3/"user": 0..5/"mpech", 4: 6..15/"127.0.0.1", 5/"host": 6..15/"127.0.0.1", 6: None, 7: None, 8: 6..15/"127.0.0.1", 9: 15..21/":repo1", 10: None, 11/"port": None, 12/"path0": None, 13: None, 14: 15..21/":repo1", 15/"path1": 16..21/"repo1", 16/"path2": None, 17: None})
[2024-02-20T17:38:12Z DEBUG pijul_remote] unknown_remote, ssh = Remote { path: "repo1", config: Config { user: "mpech", host_name: "127.0.0.1", port: 22, identity_file: None, proxy_command: None, add_keys_to_agent: No } }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] addr = "127.0.0.1" port = 22
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] identities = [Ed25519(PublicKey { key: [84, 186, 162, 173, 104, 191, 177, 10, 154, 39, 169, 95, 45, 210, 84, 214, 162, 108, 212, 193, 91, 1, 31, 78, 133, 51, 89, 26, 135, 141, 202, 245] })]
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] Trying key Ed25519(PublicKey { key: [84, 186, 162, 173, 104, 191, 177, 10, 154, 39, 169, 95, 45, 210, 84, 214, 162, 108, 212, 193, 91, 1, 31, 78, 133, 51, 89, 26, 135, 141, 202, 245] })
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] fingerprint = "xAAEjVOtHzXJgV8u6xIc4jR51ADkYivo1g/UR5I1Gkw"
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] authenticate future
[2024-02-20T17:38:12Z DEBUG libpijul::small_string] onpage_size [4, 109, 97, 105, 110]
[2024-02-20T17:38:12Z DEBUG libpijul::small_string] onpage_size [4, 109, 97, 105, 110]
[2024-02-20T17:38:12Z DEBUG pijul_remote] update_changelist_pushpull
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] run_protocol
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] waiting for a message
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 2097152 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = Success
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] run_protocol done
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] state: Id Ok("KBSPM2R64PQEEYIDHDJE5QUAD4\n")
[2024-02-20T17:38:12Z DEBUG pijul_remote] id = KBSPM2R64PQEEYIDHDJE5QUAD4
[2024-02-20T17:38:12Z DEBUG libpijul::pristine::sanakirja] last_remote: Db_ { db: 81920, k: PhantomData<sanakirja::L64>, v: PhantomData<libpijul::pristine::Pair<libpijul::pristine::hash::SerializedHash, libpijul::pristine::merkle::SerializedMerkle>>, p: PhantomData<sanakirja_core::btree::page_unsized::Page<sanakirja::L64, libpijul::pristine::Pair<libpijul::pristine::hash::SerializedHash, libpijul::pristine::merkle::SerializedMerkle>>> }
[2024-02-20T17:38:12Z DEBUG pijul_remote] dichotomy_changelist: 7 Pair { a: JNA6FZULQ7EEXF6DZAA45B2EYBZ5N2BFQ5KKN5YSWKBL64FV2TMQC, b: "U6FE4GSMJTHB4RFJPMRDAQAARAHVY4DZMXP5URX4ICM4LPTFO7UAC" }
[2024-02-20T17:38:12Z DEBUG pijul_remote] last_state: "U6FE4GSMJTHB4RFJPMRDAQAARAHVY4DZMXP5URX4ICM4LPTFO7UAC" "LBTGMZTGMZTGMZTGMZTGMZTGMZTGMZTGMZTGMZTGMZTGMZTGMZTAC"
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] get_state
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] state: State
[2024-02-20T17:38:12Z DEBUG pijul_remote] remote last_state: "U6FE4GSMJTHB4RFJPMRDAQAARAHVY4DZMXP5URX4ICM4LPTFO7UAC" "LBTGMZTGMZTGMZTGMZTGMZTGMZTGMZTGMZTGMZTGMZTGMZTGMZTAC"
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] download_changelist
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] waiting ssh, command: Ok("changelist main 8\n")
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] state changelist
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] log done
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] no msg, result = {}
[2024-02-20T17:38:12Z DEBUG pijul_remote] theirs_ge_dichotomy = []
[2024-02-20T17:38:12Z DEBUG pijul_remote] should_cache = Some(false) [] true
[2024-02-20T17:38:12Z DEBUG pijul_remote] tags = {}
[2024-02-20T17:38:12Z DEBUG libpijul::pristine::sanakirja] get_external ChangeId(3BY3ZLWR2DURO)
[2024-02-20T17:38:12Z DEBUG libpijul::pristine::sanakirja] get_external ChangeId(JNA6FZULQ7EEW)
[2024-02-20T17:38:12Z DEBUG pijul_remote] remote_has_state: "U6FE4GSMJTHB4RFJPMRDAQAARAHVY4DZMXP5URX4ICM4LPTFO7UAC"
[2024-02-20T17:38:12Z DEBUG pijul::commands::pushpull] to_upload = [Change(3BY3ZLWR2DURPX2J7UEM4GXW3OFCMNAQXB2EB7LACKN4M4KBSHPAC)]
[2024-02-20T17:38:12Z DEBUG pijul::commands::pushpull] to_upload = [Change(3BY3ZLWR2DURPX2J7UEM4GXW3OFCMNAQXB2EB7LACKN4M4KBSHPAC)]
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] upload_changes
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] Change(3BY3ZLWR2DURPX2J7UEM4GXW3OFCMNAQXB2EB7LACKN4M4KBSHPAC)
Uploading changes [==================================================] 1/1 [00:00:00] [2024-02-20T17:38:12Z DEBUG libpijul::pristine::sanakirja] commit_channel "main"
[2024-02-20T17:38:12Z DEBUG libpijul::pristine::sanakirja] Commit_channel.
[2024-02-20T17:38:12Z DEBUG libpijul::pristine::sanakirja] Commit_channel, dbs_channels = Db_ { db: 20480, k: PhantomData<libpijul::small_string::SmallStr>, v: PhantomData<libpijul::pristine::SerializedChannel>, p: PhantomData<sanakirja_core::btree::page_unsized::Page<libpijul::small_string::SmallStr, libpijul::pristine::SerializedChannel>> }
[2024-02-20T17:38:12Z DEBUG libpijul::small_string] onpage_size [4, 109, 97, 105, 110]
[2024-02-20T17:38:12Z DEBUG libpijul::small_string] onpage_size [4, 109, 97, 105, 110]
[2024-02-20T17:38:12Z DEBUG libpijul::small_string] onpage_size [4, 109, 97, 105, 110]
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] free pages pop 0x2000
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] decr_rc 0x5000 None
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] FREEING PAGE 20480 5000
[2024-02-20T17:38:12Z DEBUG libpijul::pristine::sanakirja] channels: 2c000 27000 28000 29000 13000
[2024-02-20T17:38:12Z DEBUG libpijul::small_string] writing [4, 109, 97, 105, 110]
[2024-02-20T17:38:12Z DEBUG libpijul::pristine::sanakirja] Commit_channel, self.channels = Db_ { db: 8192, k: PhantomData<libpijul::small_string::SmallStr>, v: PhantomData<libpijul::pristine::SerializedChannel>, p: PhantomData<sanakirja_core::btree::page_unsized::Page<libpijul::small_string::SmallStr, libpijul::pristine::SerializedChannel>> }
[2024-02-20T17:38:12Z DEBUG libpijul::pristine::sanakirja] commit remote KBSPM2R64PQEEYIDHDJE5QUAD4
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] free pages pop 0x3000
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] decr_rc 0x6000 None
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] FREEING PAGE 24576 6000
[2024-02-20T17:38:12Z DEBUG libpijul::pristine::sanakirja] Commit_remote, dbs_remotes = Db_ { db: 12288, k: PhantomData<libpijul::pristine::RemoteId>, v: PhantomData<libpijul::pristine::SerializedRemote>, p: PhantomData<sanakirja_core::btree::page_unsized::Page<libpijul::pristine::RemoteId, libpijul::pristine::SerializedRemote>> }
[2024-02-20T17:38:12Z DEBUG libpijul::pristine::sanakirja] put OwnedSerializedRemote { _remote: L64(81920), _rev: L64(86016), _states: L64(90112), _id_rev: L64(0), _tags: L64(94208), _path: "mpech@127.0.0.1:repo1" }
[2024-02-20T17:38:12Z DEBUG libpijul::pristine::sanakirja] write_to_page: [0, 64, 1, 0, 0, 0, 0, 0, 0, 80, 1, 0, 0, 0, 0, 0, 0, 96, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 112, 1, 0, 0, 0, 0, 0, 21, 109, 112, 101, 99, 104, 64, 49, 50, 55, 46, 48, 46, 48, 46, 49, 58, 114, 101, 112, 111, 49]
[2024-02-20T17:38:12Z DEBUG libpijul::pristine::sanakirja] Commit_remote, self.dbs.remotes = Db_ { db: 12288, k: PhantomData<libpijul::pristine::RemoteId>, v: PhantomData<libpijul::pristine::SerializedRemote>, p: PhantomData<sanakirja_core::btree::page_unsized::Page<libpijul::pristine::RemoteId, libpijul::pristine::SerializedRemote>> }
[2024-02-20T17:38:12Z DEBUG libpijul::pristine::sanakirja] 49000 4b000 4d000 4f000 8000 7000 1f000 2000 3000 24000 20000 26000 d000
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] COMMIT
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] free_db = 9000
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] free pages pop 0x4000
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] decr_rc 0x9000 None
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] FREEING PAGE 36864 9000
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] COMMIT: free_db = 0x4000
[2024-02-20T17:38:12Z DEBUG sanakirja::environment] SETTING CRC 1736114819
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] commit: unlock 1
[2024-02-20T17:38:12Z DEBUG sanakirja::environment::muttxn] /COMMIT
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 0 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 32881 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 98417 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 163953 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 229489 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 295025 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 360561 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 426097 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 491633 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 557169 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 622705 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 688241 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 753777 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 983040 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 1212529 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] extended data Ok("Error: failed to fill whole buffer\n"), 1
Error: failed to fill whole buffer
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 1376256 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = WindowAdjusted { new_size: 1507328 }
[2024-02-20T17:38:12Z DEBUG pijul_remote::ssh] msg = ExitStatus { exit_status: 1 }
[2024-02-20T17:38:12Z ERROR pijul] Error: "Remote exited with status 1"
Error: Remote exited with status 1
$
tankf33der on February 20, 2024
Blocker of 1.0 stable.
descendingnerdliness on February 20, 2024
Thank you tankf33der for characterizing this. I would not have known how. I play with a little code, but nothing this big.
descendingnerdliness on February 23, 2024
It seems that if you cheat you can get a big repo to work. After all of the changes have been added and recorded, do a plain cp -r of the repo to the server and pijul seems to push the incremental changes without complaint. It is not the size of the repo that matters, it is just the size of the push. But then, you probably already knew that.
Anyway, I am using pijul now. Thanks!
When something changes, I will help test it. I have a 2.2GB and growing repo(?) that is virtually all pdf and Word docx files in multiple directories that I would love to make available to my co-worker.
tankf33der on February 25, 2024
change subject of this discussion: [REPEATED] Push via SSH: failed to fill whole buffer
tankf33der on March 11, 2024
Please.
change subject of this discussion: [REPEATED] Push via SSH: failed to fill whole buffer
While a tiny repo produces no errors, pushing a 600MB repo to a local machine with a fresh install of Debian 11 over a tailnet results in the following
pijul push -a name@server:/extra/phd Uploading changes [==================================================] 4/4 Error: failed to fill whole buffer Error: Remote exited with status 1Yes, I am entirely new to this. Thanks