The sound distributed version control system

#410 Git export/sync

Opened by hu on April 8, 2021 Feature request
hu on April 8, 2021

Right now I really do not want to move my main development branch from github to nest: The developer mindshare is just so much bigger over there at this time!

Importing git into pijul did work fine for my small test repository. So moving data into pijul was easy and fast.

What is missing for me to use pijul more is a way to make my changes visible on github! The easiest would probably be to export a set of pijul changes in a format compatible with git. This could e.g. be a text file compatible with the git mail workflow.

The optimal approach would of course be a two way sync from pijul to git and back.

I looked into the existing git code, but that is way over my head at this time:-) Sorry, I do not expect to be able to send patches for that!

pmeunier on April 23, 2021

First, thanks for your feedback.

Unfortunately, mapping concepts in the other direction isn’t super easy, especially if we want to map them again without duplicating changes.

I agree that this may help solve the chicken-and-egg problem Pijul is currently having, where people would like to use it (you’re far from being the only one!), but are afraid they’ll lose their GitHub friends.

I’m not exactly sure how to solve this problem, my understanding is that Git is suited to some workflows and repository sizes, but not all. Pijul can do everything Git does, but also has the potential to scale to large repositories thanks to patch commutation. We’re also not too afraid of merging changes in binary files.

pmeunier added tag Feature request on April 23, 2021
Geobert on April 23, 2021

I quite understand mapping patch into commits could be hard, my view on this would be a kind of “create a snapshot from current state to be commited and push to a git repo” so I could have version released on GitHub as well.

Of course collaboration would not be an option with this, but maybe it can keep some interaction with the GitHub community

hu on April 24, 2021

I absolutely want the visibility a project gets on github. In my experience github gets about three times the interactions as gitlab, which in turn will most likely get more interactions than nest at this time. So being visible to the git world is pretty important to me!

So for me export to git is way more important than a two way sync with a git repository.

pmeunier on April 24, 2021

In this case, I believe I don’t understand what you mean by “visibility”: if you export your Pijul project to GitHub, and someone contributes to your Git project on GitHub, how do you expect this to work when you try to import that into Pijul? More specifically, after the import, what should happen the next time you export to Git?

hu on April 24, 2021

I basically want users to be able to stay on github, get recent versions via the tools they know and to use their existing workflows to consume my code.

For that to work, I want my code in github and I want development activity there to reflect the state in pijul: So changes need to be visible on github ASAP, so that users get the latest work and so that the project does not look abandoned on github.

I am fine for the much smaller group of contributors to need to use pijul and the nest! My hope is that once a user has invested enough to consider doing a contribution, reading a bit of documentation on how to push into pijul will not scare them away again.

So a way to export a set of changes from pijul to git would be a big help for me.

A full sync between git and pijul would of course be better: Each developer could choose to use either pijul or git for a project (similar in spirit to e.g. git-svn integration), but that is way harder to do I think:-)

A git export feature could also raise awareness about pijul and the nest: if lots of projects suddenly start to have “Please reports bugs via the nest” or “Please use pijul to contribute”, then some people might get curious;-)

Geobert on April 24, 2021

If I understand properly, @hu wants the same thing that I describe? We work with pijul and when we want, we create an export to github, which don’t need to have all the records, but needs to be something compatible with git so we can update the git “mirror”

hu on April 25, 2021

@Geobert: Yeap, looks like we want something very similar:-)

theduke on May 3, 2021

I agree with the comments here.

Sadly Github OS projects get much more engagement than something hosted anywhere else.

Another concern is that you would really want to preserve author information to provide proper credit to contributors on in the Github UI and repo history.

You also probably wouldn’t want to require force-pushing to the Git repo.

My ideal workflow would rely on the tagging feature. (not sure what the state is there, @pmeunier ).

The required functionality is:

  • Import commits from Git .patch files by recording each patch as a Pijul change, while preserving author information and timestamps.

    pijul git import-patch [FILEPATH]*

  • Export each change between two Pijul tags (or between the last tag and the current channel state) as Git patch:

    pijul git export-patch START_TAG..[END_TAG]?

With those two primitives you can build up nice tooling for Github imports that preserves information on both sides and doesn’t require force pushing to Git.

Eg there could even be a convenience command like pijul git sync-branch --branch PR-1.

This would amount to something like this shell script (leaving out details for readability):

# Pull the branch
git pull $BRANCH
# Export 
git format-patch --stdout $FIRST_COMMIT_IN_BRANCH.. | pijul git import-patch -
pijul tag create $NEW_TAG
git checkout main
# Somehow extract the last merged tag from the latest commit in the Git repo
LATEST_TAG=git ...
pijul git export-patch $LATEST_TAG .. $NEW_TAG | git -C $GIT_REPO apply -
git push origin main
unrelentingtech on January 23, 2022

Yes, these primitives are really what’s missing!

export-patch START_TAG..[END_TAG]?

This shouldn’t require tags though. It is quite common to do git format-patch -1 abcdef69420 to export just one commit especially when working with projects that don’t use merge requests, but rather use patch review systems like Phabricator. (For example FreeBSD and Firefox… though the latter uses hg rather than git, but git-cinnabar allows working with hg repos from git.) These projects are actually a good opportunity to use pijul since they do not ever require pushing to git remotes.