From d8f46988f5373b8233c32631df863d186bd0f598 Mon Sep 17 00:00:00 2001 From: Lars Wirzenius Date: Thu, 22 Mar 2012 14:27:23 +0000 Subject: Add documentation for branching and merging --- doc/branching-merging-systems.mdwn | 339 +++++++++++++++++++++++++++++++++++++ 1 file changed, 339 insertions(+) create mode 100644 doc/branching-merging-systems.mdwn (limited to 'doc/branching-merging-systems.mdwn') diff --git a/doc/branching-merging-systems.mdwn b/doc/branching-merging-systems.mdwn new file mode 100644 index 00000000..6fe4d590 --- /dev/null +++ b/doc/branching-merging-systems.mdwn @@ -0,0 +1,339 @@ +Branching and merging at the system level in Baserock +===================================================== + +As I write this, Baserock consists of just under 70 upstream projects, +each of which we keep in their own git repository. We need a way to +manage changes to them in a sensible manner, particularly when things +touch more than one repository. What we need is a way to do branch +and merge the whole system, across all our git repositories, with +similar ease and efficiency as what git provides for an individual +project. Where possible we need to allow the use of raw git so that +we do not constrain our developers unnecessarily. + +There are other things we will want to do across all the Baserock git +repositories, but that is outside the scope of this document, and will +be dealt with later. + +A couple of use cases: + +* I have a problem on a particular device, and want to make changes to + analyze and fix it. I need to branch the specific version of everything + that was using the system image version that the device was running. + I then want to be able to make changes to selected components and build + the system with those changes. Everything I do not explicitly touch should + stay at the same version. +* I am developing Baserock and I want to fix something, or add a new + feature, or other such change. I want to take the current newest + version of everything (the mainline development branch, whatever it + might be named), and make changes to some components, and build and + test those changes. While I'm doing that, I don't want to get random + other changes by other people, except when I explicitly ask for them + (e.g. "git pull" on an individual repository.), to avoid unnecessary + conflicts and building changes that don't affect my changes. + +In both users cases, when I'm done, I want to get my changes into the +relevant branches. This might happen by merging my changes directly, +by generating a pull request on a git server, or by generating a patch +series for each affected repository to be mailed to people who can do +the merging. + +Overview +-------- + +We want a clear, convenient, and efficient way of working with multiple +repositories and multiple projects at the same time. To manage this, +we introduce the following concepts (FIXME: naming needs attention): + +* **git repository** is exactly the same as usually with git, as are + all other concepts related to git +* **system branch** is a collection of branches in individual git + repositories that together form a particular line of development of + the whole system; in other words, given all the git repositories + that are part of Baserock, system branch `foo` consists of branch + `foo` in each git repository that has a branch with that name +* **system branch directory** contains git repositories relevant to + a system branch; it need not contain all the repositories, just the + ones that are being worked on by the user, or that the user for + other reasons have checked out +* **morph mine** is where all Morph keeps global + state and shared caches, so that creating a system branch directory + is a fairly cheap operation; all the system branch directories are + inside the morph mine directory + +As a picture: + + /home/liw/ -- user's home directory + baserock/ -- morph mine + .morph/ -- morph shared state, cache, etc + unstable/ -- system branch directory: mainline devel + morphs/ -- git repository for system, stratum morphs + magnetic-frobbles/ -- system branch directory: new feature + morphs/ -- system branch specific changes to morphs + linux/ -- ditto for linux + +To use the system branching and merging, you do the following (which we'll +cover in more detail later): + +1. Initialize the morph mine. This creates the `.morph` directory and + populates it with some initial stuff. You can have as many mines as + you want, but you don't have to have more than one, and you only + need to initialize it once. +2. Branch the system from some version of it (e.g., `master`) to work + on a new feature or bug fix. + This creates a system branch directory under the mine directory. + The system branch directory initially contains a clone of the `morphs` + git repository, with some changes specific to this system branch. + (See petrification, below.) +3. Edit one or more components (chunks) in the project. This typically + requires adding more clones of git repositories inside the system + branch directory. +4. Build, test, and fix, repeating as necessary. This requires using + git directly (not via morph) in the git repositories inside the + system branch directory. +5. Merge the changes to relevant target branches. Depending on what the + change was, it may be necessary ot merge into many branches, e.g., + for each stable release branch. + +Walkthrough +----------- + +Let's walk through what happens, making things more concrete. This is +still fairly high level, and implementation details will come later. + + morph init ~/baserock + +This creates the `~/baserock` directory if it does not exist, and then +initializes it as a "morph mine" directory, by creating a `.morph` +subdirectory. `.morph` will contain the Morph cache directory, and +other shared state between the various branches. As part of the cache, +any git repositories that Morph clones get put into the cache first, +and cloned into the system branch directories from there (probably +using hard-linking for speed), so that if there's a need for another +clone of the repository, it does not need to be cloned from a server +a second time. + + cd ~/baserock + morph branch liw/foo + morph branch liw/foo baserock/stable-1.0 + morph branch liw/foo --branch-off-system=/home/liw/system.img + +Create a new system branch, and the corresponding system branch +directory. The three versions shown differ in the starting point +of the branch: the first one uses the `master` branch in `morphs`, +the second one uses the named branch instead, and the third one +gets the SHA-1s from a system image file. + +Also, clone the `morphs` git repository inside the system branch +directory. + + cd ~/baserock/liw/foo/morphs + morph petrify base-system.morph devel-system.morph + git commit -a + +Modify the specified morphologies (or the stratum morphologies they +refer to) to nail down the references to chunk repositories to use SHA-1s +instead of branch names or whatever. The changes need to be committed +to git manually, so that the user has a chance to give a good commit +message. + +Petrification can be done by resolving the chunk references against +the current state of the git repositories, or it can be done by getting +the SHA-1s directly from a system image, or a data file. + + cd ~/baserock/liw/foo + morph edit linux + +Tell Morph that you want to edit a particular component (chunk). +This will clone the repository into the system branch directory, +at the point in history indicated by the morphologies in the +local version of `morphs`. + + cd ~/baserock/liw/foo + morph git -- log -p master..HEAD + +This allows running a git command in each git repository in a +system branch directory. Morph may offer short forms ("morph status") +as well, for things that are needed very commonly. + + cd ~/baserock/baserock/mainline + morph merge liw/foo + +This merges the changes made in the `liw/foo` branch into the +`baserock/mainline` branch. The petrification changes are automatically +undone, since they're not going to be wanted in the merge. + + cd ~/baserock + morph mass-merge liw/foo baserock/stable* + +Do the merge from `liw/foo` to every branch matching `baserock/stable*` +(as expanded by the shell). This is a wrapper around the simpler +`morph merge` command to make it easier to push a change into many +branches (e.g., a security fix to all stable branches). + + +Implementation: `morph init` +-------------- + +Usage: + + morph init [DIR] + +DIR defaults to the current working directory. If DIR is given, +but does not exist, it is created. + +* Create `DIR/.morph` and the subdirectory `cache` inside it. + + +Implementation: `morph branch` +-------------- + +Usage: + + morph branch BRANCH [COMMIT] + +This needs to be run in the morph mine directory (the one initialized +with `morph init`). + +* If `./BRANCH` as a directory exists, abort. +* Create `./BRANCH` directory. +* Clone the `morphs` repository to `BRANCH/morphs`. +* Create a new branch called `BRANCH` in morphs, based either the tip of + `master` or from `COMMIT` if given. Store the SHA-1 of the branch origin + in some way so we get at it later. + + +Implementation: `morph checkout` +-------------- + +Usage: + + morph checkout BRANCH + +This needs to be run in the morph mine directory. It works like +`morph branch`, except it does not create the new branch and requires +it to exist instead. + +* If `./BRANCH` as a directory exists, abort. +* Create `./BRANCH` directory. +* Clone the `morphs` repository to `BRANCH/morphs`. +* Run `git checkout BRANCH` in the `morphs` repository. + + +Implementation: `morph petrify` +-------------- + +Usage: + + morph petrify [MORPH]... + morph petrify --petrify-from-system FILE + +This needs to be run in the `morphs` git repository in a system branch. + +In the first form: + +* read each of the given morphologies; if the morphology is a system one, + follow references to stratum morphologies and process those instead +* in each stratum morphology, replace a reference to a chunk with the + absolute SHA-1: if the original reference was, say, `baserock/morph`, + get the SHA-1 of the current tip commit in that branch and replace + the reference in the morphology + +In the second form: + +* extract the system and stratum morphologies used in the system image file; + these are in a petrified form already +* copy the morphologies to the current working directory, overwriting the + files from git + +In either case, the results need to be committed (with normal git commands) +by the user. + + +Implementation: `morph edit` +-------------- + +Usage: + + morph edit REPO MORPH... + +where `REPO` is a chunk repository (absolute URL or one relative to one of +the `git-base-url` values). The command must be run in the `morphs` +directory of the system branch. + +* `git clone REPOURL` where the URL is constructed with `git-base-url` + if necessary. +* `git branch BRANCH REF` where `BRANCH` is the branch name given to + `morph branch` and `REF` is the reference to the chunk we want to edit, + as specified in morphologies. +* Modify the affected morphologies to refer to the repository using + the `BRANCH` name, and commit those changes. + +If the specified morphology is not a stratum morphology (that is, it is +a system one), we check all the stratum morphologies mentioned and find +the one that refers to the specified repository. + +Multiple morphologies can be specified. They must have the same original +reference to the repository. However, they will all be modified. + + +Implementation: `morph git` +-------------- + +Usage: + + morph git -- log -p master..HEAD + +This is to be run in the morph mine. It runs git with the arguments on +the command line in each local git repository in the mine. (The `--` is +only necessary if the git arguments are to contain options.) + + +Implementation: `morph merge` +-------------- + +Usage: + + morph merge BRANCH + +This needs to be run inside a system branch directory, and `BRANCH` +must be another system branch checked out in the morph mine. + +* In each git repository modified by the `BRANCH` system branch, + run `git merge --no-commit BRANCH`, then undo any changes to + stratum morphologies made by `morph edit`, and finally commit + the changes. + + +Implementation: `morph mass-merge` +-------------- + +Usage: + + morph mass-merge BRANCH [TARGET]... + +To be run in the morph mine directory. + +This just runs `morph merge BRANCH` in each `TARGET` system branch. + + +Implementation: `morph cherry-pick` +-------------- + +Usage: + + morph cherry-pick BRANCH [COMMIT]... + morph cherry-pick BRANCH --since-branch-point + +To be run in the system branch directory. + +In the first form: + +* For each git repository modified by the `BRANCH` system branch, + run `git cherry-pick COMMIT` for each `COMMIT`. + +In the second form: + +* For each git repository modified by the `BRANCH` system branch, + run `git cherry-pick` giving it a list of all commits made after + the system branch was created. + -- cgit v1.2.1