Experimenting with Pijul 1.0.0-beta.2 running on Linux, managing Dos/Win CRLF terminated files (aka \r\n, CarriageReturn Linefeed).

I progressed into a highly broken state, which might be separate bugs, or probably all related to CRLF issues.

A problem seems to be triggered when Pijul has to Output the files, for example when switching channels. The files are then created with only Newlines (\n), instead of CRLF. At this point the repo and working dir are effectively wholly broken. The working dir looks like it has completely changed to Pijul.

Furthermore, I got in a state where it was ignoring the most recent Change (s ?) So Pijul already thinks there is a huge delta with all the lines (because of the CRLF issue), and the delta is actually against the version one (or more?) Patch backwards, like it is ignoring the most recent one(s).

Further, with other tests, I got in a state where it just keeps Record-ing the same change repeatedly. Make a 1-letter change to a CRLF file, repeatedly do pijul diff pijul record You see the diff, edit and save the Record, but it never “takes”, just keeps re-creating the same change in the repo, so the log shows a whole series of the same 1-letter change.

It seems there are a couple different issues/effects, though maybe they are all the same underlying bug.

1. CRLF is not correctly recognized/handled in the change detection, and/or patch storage.

2. CRLF is not correctly Output, when Pijul has to create a file from scratch.

I have debugged this to some reproducible test cases if you need more details. But I think it should be reproducible with any CRLF files – operations rapidly fail in a variety of ways.

I found this other discussion which sounds related, with some patches: #708 Lenient EOL in record change parsing

That report is terse and I can’t tell if it’s the exact same issues as here. I started testing a private ‘cargo build’ with that patch, but encountered some other problem(s), therefore stopped investigating as I’m not sure if that Patch is comprehensive, and whether it applies on the 1.0.0-beta.2 release or to the current development head.

Overall, I recommend Pijul to correctly handle all common line-terminator standards. At least, Unix: LF (Newline, \n) Dos/Windows: CRLF (\r\n) and if possible Mac legacy: CR (\r)

It must detect them in the input files for computing deltas, and correctly reproduce when Outputting the files (and any other purposes). It should work with all these standards on all the supported operating platforms, at the distinct file level, allow different format files in the same repository, and safely manipulate a repository of mixed standard files from any platform.

I suggest that be a release requirement for 1.0.

#708 was an attempt to fix the EOL issues discussed here https://pijul.zulipchat.com/#narrow/stream/270693-general/topic/CRLF.20line.20endings

Thanks for the report. Now that #708 is closed (and the patch has been applied to main), @jbthiel do you have a scenario that reproduces the issue consistently?