Rationale
-
See tails#8654 (closed).
-
Our automated builds/testing/QA/CI/whatever system needs that.
-
For example, it would be good to allow building Jessie-based topic branches with the
feature-jessie
APT suite. Right now, the only way to build a topic branch based on Jessie is to first merge it intofeature/jessie
locally... and our Jenkins stuff doesn't support that, so we cannot add Jessie-based topic branches to our automatic ISO build setup yet.
Random ideas and notes
Storing the base branch's name elsewhere
... e.g. in config/base_branch
, as opposed to recording the
corresponding base APT suite in config/APT_overlays
?
This may be useful for other purposes: how CI infrastructure will soon need to know what's the base branch for a given topic branch.
Current conclusion: Yes. The work done in
feature/8654-encode-apt-suite-in-git
assumes we'll be doing that.
(Update: we finally went for config/APT_overlays.d/*
, implemented in
feature/8654-encode-apt-suite-in-git-with-APT-overlays-dot-d
.)
Also, it could be needed if/once we use the union merge driver for
config/APT_overlays
(when merging a base branch into another one, the
union merge driver wouldn't DTRT, and surely another merge driver
would be more suitable for config/base_branch
; that merge driver
should ensure that the current version of the file is left as is,
which can possibly be achieved simply with driver = /bin/true
).
Getting rid of the base APT suites
Entirely get rid of the suites for the main branches: only keep a base
branch per base-major-distro-version (tails-wheezy
, tails-jessie
,
etc.)?
Once we stop adding packages to main branches' APT suites except at
release time, it make little sense to merge their pile of overlays
into each base suite when we release, while we can simply merge them
into e.g. tails-wheezy
, and in turn into the release tag's APT
suite. If we go this way, we need to encode the distribution name
somewhere in Git, e.g. config/Debian_codename
. As a bonus, if we do
that, we could even:
- pass the content of that file to
$RUN_LB_CONFIG --distribution
inauto/config
- possibly generate parts of
config/chroot_sources
andconfig/chroot_apt/preferences
dynamically at build time - in the end, we could almost build ISOs based on Debian sid simply by changing a single configuration file in Git?
Current conclusion: No. It makes things a little bit too
complicated e.g. during freeze time for major releases: one wonders
where the overlay APT suites merged into testing
would
be aggregated.
gitattributes
gitattributes(5)
files
enable us to define path-specific behaviour, such as a speficic merge
driver, or filters that shall be applied upon checkin or checkout.
For what matters here, this file should be stored in .gitattributes
at the root of our main Git repository; whatever is defined in-tree
can be overriden locally via $GIT_DIR/info/attributes
. This means
that we can have:
- default settings suitable for human contributors, stored in
.gitattributes
in Git; - local overrides for bits of our infrastructure that may need to behave differently.
Git ships with some built-in filters and merge drivers, that can
directly be used by specifying what path they should apply to via
gitattributes(5)
. However, it's a different matter whenever one
wants to use custom merge drivers and filters: they have to be
defined in Git configuration, which can't be stored in the repo
itself. Then, contributors who need their Git to use custom filters
and merge drivers have to add an [include]
section to their
.git/config
, that points to a config file:
- that defines the needed filters and merge drivers;
- that is hosted in another repo: otherwise, when checking out an
untrusted branch, arbitrary untrusted code could be run e.g.
via a
smudge
filter.
Note that the merge driver only "affects how three versions of a file are merged when a file-level merge is necessary during git merge". It's not going to be used at all if only one side of the merge has modified the file that has a merge driver defined.
config/APT_overlays
Merge driver for What merge driver should we use for config/APT_overlays
?
What follows assumes that we're not storing the base branch's APT suite in that file.
-
When doing base-branch-into-topic-branch merges in Jenkins, we'll probably want to use the union merge driver for
config/APT_overlays
, in order to resolve trivial merge conflicts by including the lines from both sides. Ideally, we should use the union merge driver only when the Git merge strategy being used would leave a conflict unresolved. To achieve this, it might be that we can use a custom merge driver, that first tries runninggit merge-file
, and if that one returns a strictly positive number, then runs it again with--union
. -
When manually merging a base branch into another one, the union merge driver is not very good: it'll introduce duplicated lines in
config/APT_overlays
, e.g. when merging e.g.stable
intodevel
. Aclean
filter defined in.gitattributes
might be enough to de-duplicate these lines. -
When manually merging a base branch into a topic branch, there are several sub-cases:
- the base branch's
config/APT_overlays
has had new suites added: then we want to add them in the topic branch too; in this case the union merge driver could work, except if the base branch'sconfig/APT_overlays
also had lines removed during the last release; - the base branch's
config/APT_overlays
has had suites removed, e.g. during a release: then, ideally we want to remove them from the topic branch as well. It's no big deal in practice if lines that were deleted fromconfig/APT_overlays
in the base branch are still in the topic branch post-merge (the same packages are in the base branch's APT suite anyway), but it's a bit ugly and there are chances that we end up mistakenly re-adding the deleted lines inconfig/APT_overlays
in the base branch when merging the topic branch in it, and then the release manager would merge these APT suites into the corresponding base APT suite at release time, potentially downgrading packages that have been upgraded by other merged branches.
- the base branch's
At this point, it doesn't feel worth analyzing all other possible cases. It seems clear that no custom merge driver would do the right thing in all cases of merges we do manually.
Current conclusion:
- human contributors will handle
config/APT_overlays
the good old way, as any other file in our Git tree; - we'll use the union merge driver in Jenkins.
Update: all this reasoning is moot since we've moved to
config/APT_overlays.d/*
, so we don't need the union merge
driver anymore.
And later, we can reconsider if we need something more clever, e.g.:
- We might have to use a custom merge driver to force a manual
conflict resolution for every manual merge that involves
config/APT_overlays
, to avoid the case when Git tries to be clever and makes the wrong decision. - For Jenkins merges, we may need to investigate using the union merge driver only when it's really needed.
config/base_branch
Merge driver for What merge driver should we use for config/base_branch
?
When merging between branches that have the same base branch, there's no conflict, what should be done is quite obvious and then Git will figure it out just fine.
When merging between branches that don't have the same base branch, the cases that matter most are:
-
Merging a base branch (B1) into another one (B2): as above,
config/base_branch
should be left unchanged in the resulting B2. -
Changing the base branch of a topic branch T, from B1 to B2: we want T's old base branch to be replaced by B2.
-
Merging into B2 a topic branch T which is based on B1. In practice, this should only happen when B1 = stable or testing, and B2 = devel, and then the Right Way™ to do it would instead be to:
- either first merge T into B1 if it's ready, and then B1 into B2: the first step is trivial, and the second one is the "merging a base branch into another one" case, which was covered above;
- or, merge B2 into T, and then T into B2: the first step is
essentially about "changing the base branch of a topic branch"
case, which was covered above; the second step is trivial.
But mistakes may arise, so ideally we would cover cases when merges
are not done in ideal order. What should happen, then, is that
config/base_branch
is left unchanged in B2 after the merge.
To sum up, in all cases except "changing the base branch of a topic
branch", we want to keep whatever was in the branch we're merging
into. To achieve the general case, perhaps one can define a custom
merge driver, used for config/base_branch
, that would simply call
git merge-file --ours
if we're merging into a base branch, and git merge-file
otherwise. Git will handle the "changing the base branch
of a topic branch" corner case somehow, and it's still the
responsibility of the topic branch's developer to ensure that
config/base_branch
is correct.
Current conclusion: all this feels like premature optimization, or bonus foolproof protection. We'll see how it goes in practice.