1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
|
Branching and merging at the system level in Baserock
=====================================================
NOTE: This is a spec. The code does not yet match it.
As I write this, Baserock consists of just under 70 upstream projects,
each of which we keep in their own git repository. We need a way to
manage changes to them in a sensible manner, particularly when things
touch more than one repository. What we need is a way to do branch
and merge the whole system, across all our git repositories, with
similar ease and efficiency as what git provides for an individual
project. Where possible we need to allow the use of raw git so that
we do not constrain our developers unnecessarily.
There are other things we will want to do across all the Baserock git
repositories, but that is outside the scope of this document, and will
be dealt with later.
A couple of use cases:
* I have a problem on a particular device, and want to make changes to
analyze and fix it. I need to branch the specific version of everything
that was using the system image version that the device was running.
I then want to be able to make changes to selected components and build
the system with those changes. Everything I do not explicitly touch should
stay at the same version.
* I am developing Baserock and I want to fix something, or add a new
feature, or other such change. I want to take the current newest
version of everything (the mainline development branch, whatever it
might be named), and make changes to some components, and build and
test those changes. While I'm doing that, I don't want to get random
other changes by other people, except when I explicitly ask for them
(e.g. "git pull" on an individual repository.), to avoid unnecessary
conflicts and building changes that don't affect my changes.
In both users cases, when I'm done, I want to get my changes into the
relevant branches. This might happen by merging my changes directly,
by generating a pull request on a git server, or by generating a patch
series for each affected repository to be mailed to people who can do
the merging.
Overview
--------
We want a clear, convenient, and efficient way of working with multiple
repositories and multiple projects at the same time. To manage this,
we introduce the following concepts (FIXME: naming needs attention):
* **git repository** is exactly the same as usually with git, as are
all other concepts related to git
* **system branch** is a collection of branches in individual git
repositories that together form a particular line of development of
the whole system; in other words, given all the git repositories
that are part of Baserock, system branch `foo` consists of branch
`foo` in each git repository that has a branch with that name
* **system branch directory** contains git repositories relevant to
a system branch; it need not contain all the repositories, just the
ones that are being worked on by the user, or that the user for
other reasons have checked out
* **morph workspace** is where all Morph keeps global
state and shared caches, so that creating a system branch directory
is a fairly cheap operation; all the system branch directories are
inside the morph workspace directory
As a picture:
/home/liw/ -- user's home directory
baserock/ -- morph workspace
.morph/ -- morph shared state, cache, etc
unstable/ -- system branch directory: mainline devel
morphs/ -- git repository for system, stratum morphs
magnetic-frobbles/ -- system branch directory: new feature
morphs/ -- system branch specific changes to morphs
linux/ -- ditto for linux
To use the system branching and merging, you do the following (which we'll
cover in more detail later):
1. Initialize the morph workspace. This creates the `.morph` directory and
populates it with some initial stuff. You can have as many workspaces as
you want, but you don't have to have more than one, and you only
need to initialize it once.
2. Branch the system from some version of it (e.g., `master`) to work
on a new feature or bug fix.
This creates a system branch directory under the workspace directory.
The system branch directory initially contains a clone of the `morphs`
git repository, with some changes specific to this system branch.
(See petrification, below.)
3. Edit one or more components (chunks) in the project. This typically
requires adding more clones of git repositories inside the system
branch directory.
4. Build, test, and fix, repeating as necessary. This requires using
git directly (not via morph) in the git repositories inside the
system branch directory.
5. Merge the changes to relevant target branches. Depending on what the
change was, it may be necessary ot merge into many branches, e.g.,
for each stable release branch.
Walkthrough
-----------
Let's walk through what happens, making things more concrete. This is
still fairly high level, and implementation details will come later.
morph init ~/baserock
This creates the `~/baserock` directory if it does not exist, and then
initializes it as a "morph workspace" directory, by creating a `.morph`
subdirectory. `.morph` will contain the Morph cache directory, and
other shared state between the various branches. As part of the cache,
any git repositories that Morph clones get put into the cache first,
and cloned into the system branch directories from there (probably
using hard-linking for speed), so that if there's a need for another
clone of the repository, it does not need to be cloned from a server
a second time.
cd ~/baserock
morph branch liw/foo
morph branch liw/foo baserock/stable-1.0
morph branch liw/foo --branch-off-system=/home/liw/system.img
Create a new system branch, and the corresponding system branch
directory. The three versions shown differ in the starting point
of the branch: the first one uses the `master` branch in `morphs`,
the second one uses the named branch instead, and the third one
gets the SHA-1s from a system image file.
Also, clone the `morphs` git repository inside the system branch
directory.
cd ~/baserock/liw/foo/morphs
edit base-system.morph devel-system.morph
git commit -a
Modify the specified morphologies (or the stratum morphologies they
refer to) to nail down the references to chunk repositories to use SHA-1s
instead of branch names or whatever. The changes need to be committed
to git manually, so that the user has a chance to give a good commit
message.
Petrification is useful to prevent the set of changes including changes
by other team members. When a chunk is edited it will be made to refer
to that ref instead of the SHA-1 that it is petrified to.
Petrification can be done by resolving the chunk references against
the current state of the git repositories, or it can be done by getting
the SHA-1s directly from a system image, or a data file.
cd ~/baserock/liw/foo
morph edit linux
Tell Morph that you want to edit a particular component (chunk).
This will clone the repository into the system branch directory,
at the point in history indicated by the morphologies in the
local version of `morphs`.
cd ~/baserock/liw/foo
morph git -- log -p master..HEAD
This allows running a git command in each git repository in a
system branch directory. Morph may offer short forms ("morph status")
as well, for things that are needed very commonly.
cd ~/baserock/baserock/mainline
morph merge liw/foo
This merges the changes made in the `liw/foo` branch into the
`baserock/mainline` branch. The petrification changes are automatically
undone, since they're not going to be wanted in the merge.
cd ~/baserock
morph mass-merge liw/foo baserock/stable*
Do the merge from `liw/foo` to every branch matching `baserock/stable*`
(as expanded by the shell). This is a wrapper around the simpler
`morph merge` command to make it easier to push a change into many
branches (e.g., a security fix to all stable branches).
Implementation: `morph init`
--------------
Usage:
morph init [DIR]
DIR defaults to the current working directory. If DIR is given,
but does not exist, it is created.
* Create `DIR/.morph`.
Implementation: `morph branch`
--------------
Usage:
morph branch BRANCH [COMMIT]
This needs to be run in the morph workspace directory (the one initialized
with `morph init`).
* If `./BRANCH` as a directory exists, abort.
* Create `./BRANCH` directory.
* Clone the `morphs` repository to `BRANCH/morphs`.
* Create a new branch called `BRANCH` in morphs, based either the tip of
`master` or from `COMMIT` if given. Store the SHA-1 of the branch origin
in some way so we get at it later.
Implementation: `morph checkout`
--------------
Usage:
morph checkout BRANCH
This needs to be run in the morph workspace directory. It works like
`morph branch`, except it does not create the new branch and requires
it to exist instead.
* If `./BRANCH` as a directory exists, abort.
* Create `./BRANCH` directory.
* Clone the `morphs` repository to `BRANCH/morphs`.
* Run `git checkout BRANCH` in the `morphs` repository.
Implementation: `morph edit`
--------------
Usage:
morph edit REPO MORPH...
where `REPO` is a chunk repository (absolute URL or one relative to one of
the `git-base-url` values). The command must be run in the `morphs`
directory of the system branch.
* `git clone REPOURL` where the URL is constructed with `git-base-url`
if necessary.
* `git branch BRANCH REF` where `BRANCH` is the branch name given to
`morph branch` and `REF` is the reference to the chunk we want to edit,
as specified in morphologies.
* Modify the affected morphologies to refer to the repository using
the `BRANCH` name, and commit those changes.
If the specified morphology is not a stratum morphology (that is, it is
a system one), we check all the stratum morphologies mentioned and find
the one that refers to the specified repository.
Multiple morphologies can be specified. They must have the same original
reference to the repository. However, they will all be modified.
Implementation: `morph git`
--------------
Usage:
morph git -- log -p master..HEAD
This is to be run in the morph workspace. It runs git with the arguments on
the command line in each local git repository in the workspace. (The `--` is
only necessary if the git arguments are to contain options.)
Implementation: `morph merge`
--------------
Usage:
morph merge BRANCH
This needs to be run inside a system branch directory's `morphs`
repository, and `BRANCH` must be another system branch checked out
in the morph workspace.
* In each git repository modified by the `BRANCH` system branch,
run `git merge --no-commit BRANCH`, then undo any changes to
stratum morphologies made by `morph edit`, and finally commit
the changes.
Implementation: `morph mass-merge`
--------------
Usage:
morph mass-merge BRANCH [TARGET]...
To be run in the morph workspace directory.
This just runs `morph merge BRANCH` in each `TARGET` system branch.
Implementation: `morph cherry-pick`
--------------
Usage:
morph cherry-pick BRANCH [COMMIT]...
morph cherry-pick BRANCH --since-branch-point
To be run in the system branch directory.
In the first form:
* For each git repository modified by the `BRANCH` system branch,
run `git cherry-pick COMMIT` for each `COMMIT`.
In the second form:
* For each git repository modified by the `BRANCH` system branch,
run `git cherry-pick` giving it a list of all commits made after
the system branch was created.
|