custom.mdwn 13.9 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[[!meta title="Custom APT repository"]]

We use a custom APT repository to store our custom packages.

[[!toc levels=2]]

Overview
========

We use one single APT repository hosting multiple *suites*:

* We have a (read-only) suite for every past release: `0.9`,
  `0.10.1`, etc.
* We have a suite for each *main* branch: `stable`, `testing`,
  `devel`, `feature-jessie`
* We have an overlay suite for each *topic* branch: `bugfix/*`,
  `feature/*`, etc.
  **Note**: the APT suite corresponding to a given Git topic
  branch contains *only* the packages this branch adds to the tag or
  *main* branch it diverged from. Think of it as an overlay.
* We also have a less formal `unstable` suite, that should not be used
  by any Tails git branch; it can be used as hosting space for other
  packaging work we might do, e.g. acting as upstream or
  Debian maintainers.
25
26
* We also have a `builder-jessie` suite, used to provide additional
  packages needed on a Jessie system to build Tails.
27
28
29
30
31
32

The suite(s) to use as sources for APT, during the build and inside
the resulting system, are determined by the content of the
`config/base_branch` and `config/APT_overlays.d/*` files. See details in
the *Build system* section below.

33
34
35
Basically, a cronjob fetches the Tails Git repository every
few minutes, detects new branches, and accordingly create new suites
in the custom APT repository.
36

37
38
39
40
41
42
We manage our APT repository with
[reprepro](http://mirrorer.alioth.debian.org/).
See the corresponding
[[documentation|contribute/working_together/roles/sysadmins#custom-apt-repository]]
for details.

43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
Build system
============

The Tails ISO build system dynamically adds APT sources that will be
used during the build, and inside the resulting ISO itself.

If the last version in `debian/changelog` was released already (i.e.
a matching tag exists), then the build system adds the suite
corresponding to this release (e.g. `1.5` or `3.0`), and that's all.

Else, it adds:

* one APT source for the base branch of the one being built, as found
  in `config/base_branch`;
* one APT source for each suite listed in
  `config/APT_overlays.d/*`; note that only the name of such
  files matters, and their content is ignored.

intrigeri's avatar
intrigeri committed
61
In practice, `config/APT_overlays.d/` should contain:
62
63
64
65

* for a topic branch:
  - if needed, a file that is named like the branch's own overlay APT
    suite; e.g. for the `bugfix/12345-whatever` branch, it would be
intrigeri's avatar
intrigeri committed
66
    called `config/APT_overlays.d/bugfix-12345-whatever`
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
  - any file representing APT suites that came from merging its base
    branch into this topic branch, that is:
* for a base branch (`stable`, `testing`, `devel` or
  `feature/jessie`): a file for each additional, overlay APT suite that
  came from topic branches that ship Debian packages and were merged
  into this base branch since last time it was used to prepare
  a release.

The code that implements this is [[!tails_gitweb
auto/scripts/tails-custom-apt-sources]]. It has [[!tails_gitweb
features/build.feature desc="automated tests"]].

At release time, the release manager:

1. merges into the release branch's APT suite all APT overlay
   suites found in `config/APT_overlays.d/`;
2. empties `config/APT_overlays.d/` in the release branch;
3. merges the release branch into other base branches as needed, and
   ensures that all resulting `config/APT_overlays.d/`:s make sense.

87
Note that a branch like `feature/jessie` needs to be a base branch: we want to be
88
89
90
91
92
able to work on topic branches forked off `feature/jessie`.

SSH access
==========

93
Configure your SSH client to connect to the APT server:
94
95
96
97
98
99
100

	Host incoming.deb.tails.boum.org
		Port 3003

HTTP access
===========

101
The custom APT repository can be browsed at <https://deb.tails.boum.org/>.
102
103
104
105
106
107
108
109

Workflow
========

Creating a new branch
---------------------

Push your branch to Git and wait a few minutes for the new APT suite
intrigeri's avatar
intrigeri committed
110
111
112
113
114
115
to appear on <https://deb.tails.boum.org/dists/>. You can look up the
name of that suite there.

Then, you probably want to drop a new file in
`config/APT_overlays.d/`, named after the APT suite corresponding to
your new branch. See details in the *Build system* section above.
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145

Importing a new package
-----------------------

### Building a package

Make sure the `Distribution:` field in your `.changes` file matches
the suite you want the package to land in (e.g.
pass `--changes-option=-DDistribution=feature-torbrowser` to
pdebuild's `--debbuildopts`).

Make sure to have the `.changes` file include the original source
archive (`.orig.tar.{gz,bz2,xz}`) if it is not already in our APT
repository; this can be done by passing `-sa` to pdebuild's
`--debbuildopts`.

### Configuring an upload tool

#### Configuring dupload

Add this configuration snippet to your `dupload` configuration:

	$config::cfg{'tails'} = {
	        fqdn => "incoming.deb.tails.boum.org",
	        method => "scp",
	        login => "reprepro",
	        incoming => "/srv/reprepro/incoming/",
	        dinstall_runs => 1,
	};

146
#### Configuring dput
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166

Add this to `.dput.cf`:

    [tails]
    fqdn            = incoming.deb.tails.boum.org
    method          = scp
    login           = reprepro
    incoming        = /srv/reprepro/incoming/
    run_dinstall    = 0


### Uploading and importing process

Carefully prepare and build your package. Usual precautions, (Lintian
etc.) apply.

Carefully check the `.changes` file (especially the `Distribution`
control field, and the included files list; the former can be fixed
with the `changestool(1)` command, from [[!debpkg reprepro]]).

167
Remove all `buildinfo` and `dbgsym` lines from the `.changes` file.
168
169
XXX: still needed? We now run reprepro from Stretch, that supports
both in theory.
170

intrigeri's avatar
intrigeri committed
171
172
If the `.orig.tar.{gz,bz2,xz}` tarball is present neither in the
`.changes` file nor in our custom APT repository, add it using:
173
174
175

	$ changestool $CHANGES_FILE includeallsources

intrigeri's avatar
intrigeri committed
176
177
178
179
Make sure that the `.changes` file lists the _binary_ packages too:
Tails has no package auto-builder that would build them from source
for you.

180
181
182
183
184
185
186
187
Sign the `.changes` file with a key that is in the uploaders list:

	$ debsign $CHANGES_FILE

Upload the files to the incoming queue:

	$ dupload --to tails $CHANGES_FILE

Ulrike Uhlig's avatar
Ulrike Uhlig committed
188
189
190
191
or, using dput:

	$ dput tails $CHANGES_FILE

192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
reprepro will automatically notice the new files and import them into
the suite specified in your `.changes` file.

Check the result:

	$ ssh reprepro@incoming.deb.tails.boum.org reprepro list $SUITE $PACKAGENAME

<a id="workflow-merge-main-branch"></a>

Merging a main branch
----------------------

When a Git *main* branch (`devel`, `testing`, `stable`,
`feature/jessie`) is merged into another *main* branch, the corresponding
operation must be done on the APT suites.

1. Save the list of packages currently present in the APT suite we
   want to merge *into*, e.g. `reprepro list devel`.

2. Make sure you are not going to overwrite newer packages with
   older ones (hint: use the `tails-diff-suites` script).

3. Merge the APT suites:

   1. Set some environment variables:

218
219
220
221
            # the branch you want to merge
            SRC=stable
            # the branch you want to merge _into_
            DST=devel
222
223
224

   2. Merge in Git and APT:

225
226
227
228
            git checkout "$DST" && \
            git merge "$SRC" && \
            ssh reprepro@incoming.deb.tails.boum.org \
                 tails-merge-suite "$SRC" "$DST"
229
230
231

   3. Restore the `config/base_branch` if needed:

232
233
            echo "${DST}" > config/base_branch && \
            git commit config/base_branch -m "Restore ${DST}'s base branch." || :
234
235
236

   4. Push:

237
            git push origin "${DST}:${DST}"
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314

4. Make sure not to re-add, into the branch we merge into, any package
   that was removed from it, but still is in the branch we merge from:
   e.g. when merging `stable` into `devel`, it may be that
   `devel` had some packages removed (e.g. due to previously
   merging a topic branch into it, whose purpose is to *remove* custom
   packages). To this end, compare the resulting list of (package,
   version) in the `devel` APT suite with the one saved before
   the merge, check Git
   merges history if needed, apply common sense, and remove from
   `devel` the packages that were removed from it a while ago,
   and were just erroneously re-added by the merge operation.

<a id="workflow-reset"></a>

Resetting a suite to the state of another one
---------------------------------------------

   a. First, set some environment variables:

        # the suite to reset
        OLD=testing
        # the final state it should be in
        NEW=devel

   b. Then, empty the `OLD` suite:

      	ssh reprepro@incoming.deb.tails.boum.org \
      	      reprepro removematched $OLD '\*'

   c. Finally, merge `NEW` into `OLD`

      	ssh reprepro@incoming.deb.tails.boum.org \
      	      tails-merge-suite $NEW $OLD

<a id="workflow-merge-overlays"></a>

Merging APT overlays
--------------------

This operation merges all APT overlays listed in the given branch's
`config/APT_overlays.d/` into its own APT suite, empties
`config/APT_overlays.d/` accordingly, then commits and pushes to Git.

1. Set some environment variables:

        # The branch that should have its overlays merged
        BRANCH=devel

2. Merge the APT overlays in reprepro:

        git checkout "$BRANCH" && \
        for overlay in $(ls config/APT_overlays.d/) ; do
           if ! ssh reprepro@incoming.deb.tails.boum.org \
                   tails-merge-suite "$overlay" "$BRANCH" ; then
              echo "Failed to merge '$overlay' into '$BRANCH': $?" >&2
              break
           fi
        done

3. Empty `config/APT_overlays.d/`:

        git checkout "$BRANCH" && \
        git rm config/APT_overlays.d/* && \
        git commit config/APT_overlays.d/ \
           -m "Empty the list of APT overlays: they were merged"

4. Push the Git branch:

        git push origin "${BRANCH}:${BRANCH}"

<a id="workflow-post-tag"></a>

Tagging a new Tails release
---------------------------

Once the new release's Git tag is pushed, a cronjob creates
315
a new APT suite on the custom APT repository's side within a few minutes.
316
317
This new APT suite is called the same as the new release version.

intrigeri's avatar
intrigeri committed
318
319
320
Wait for this new (empty) APT suite to be created and initialize it
with the packages currently found in the APT suite corresponding to
the branch that is used to prepare the release:
321

intrigeri's avatar
intrigeri committed
322
323
324
325
	while ! ssh reprepro@incoming.deb.tails.boum.org reprepro list "${TAG:?}" >/dev/null 2>&1; do
	   sleep 5
	done && \
	ssh reprepro@incoming.deb.tails.boum.org \
326
327
328
329
330
331
332
	     tails-merge-suite "$RELEASE_BRANCH" "$TAG"

<a id="workflow-post-release"></a>

After a new Tails release is out
--------------------------------

333
### If you just put out a final release
334

335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
If the release was a major one, then:

1. [[Hard reset the `stable` APT suite to
   the state of the `testing` one|APT_repository/custom#workflow-reset]].

2. Empty `config/APT_overlays.d` in the `stable` branch:

        git checkout stable && \
		if [ $(find config/APT_overlays.d -maxdepth 1 -type f | wc -l) -gt 1 ]; then
           git rm config/APT_overlays.d/* && \
           git commit config/APT_overlays.d/ \
              -m "Empty the list of APT overlays: they were merged"
        fi

In any case:

351
352
* [[merge `stable` or `testing` into
  `devel`|APT_repository/custom#workflow-merge-main-branch]]
intrigeri's avatar
intrigeri committed
353
* increment the version number in `devel`'s `debian/changelog` to match
354
355
  the next major release, so that
  next builds from the `devel` branch do not use the APT suite meant
intrigeri's avatar
intrigeri committed
356
357
358
359
360
361
362
363
364
  for the last release:

        cd "${RELEASE_CHECKOUT}" && \
        git checkout devel && \
        dch --newversion "${NEXT_PLANNED_MAJOR_VERSION:?}" \
           "Dummy entry for next release." && \
        git commit debian/changelog \
           -m "Add dummy changelog entry for ${NEXT_PLANNED_MAJOR_VERSION:?}."

365
366
367
* increment the version number in stable's `debian/changelog` to match
  the next point release, so that
  next builds from the `stable` branch do not use the APT suite meant
intrigeri's avatar
intrigeri committed
368
369
370
371
372
373
374
375
376
  for the last release:

        cd "${RELEASE_CHECKOUT}" && \
        git checkout stable && \
        dch --newversion "${NEXT_PLANNED_MINOR_VERSION:?}" \
           "Dummy entry for next release." && \
        git commit debian/changelog \
           -m "Add dummy changelog entry for ${NEXT_PLANNED_MINOR_VERSION:?}."

377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
### Else, if you just released a RC

* increment the version number in `debian/changelog` on the branch
  used for the release, to match the upcoming non-RC release, so that
  the next builds from it do not use the APT suite meant for the RC:

        cd "${RELEASE_CHECKOUT}" && \
        git checkout "${RELEASE_BRANCH:?}" && \
        dch --newversion "${NEXT_PLANNED_MAJOR_VERSION:?}" \
           "Dummy entry for next release." && \
        git commit debian/changelog \
           -m "Add dummy changelog entry for ${NEXT_PLANNED_MAJOR_VERSION:?}."

* increment the version number in `devel`'s `debian/changelog` to
  match the second next major release, so that images built from there
  have the right version number:

        cd "${RELEASE_CHECKOUT}" && \
        git checkout devel && \
        dch --newversion "${SECOND_NEXT_PLANNED_MAJOR_VERSION:?}" \
           "Dummy entry for next release." && \
        git commit debian/changelog \
           -m "Add dummy changelog entry for ${SECOND_NEXT_PLANNED_MAJOR_VERSION:?}."

401
402
403
404
Giving access to a core developer
---------------------------------

1. Give SSH access to the `reprepro` user on the system that hosts
405
406
   reprepro (using the `ssh_authorized_key` Puppet resource, until
   [[!tails_ticket 12346]] is resolved).
407
408
409
410
411
412
413
414
415
416
417
418
419
2. Import the developer's public GnuPG key into the `reprepro` user's
   GnuPG keyring -- should be doable using Puppet, some day
3. Add the developer's OpenPGP key ID to `$reprepro_uploaders` in our
   `tails::reprepro` Puppet module. Deploy.

Contributing without privileged access
--------------------------------------

Non-core developers without access to the "private" APT infrastructure
would add the .deb they want to their Git branch as we have been
doing until now, push the result on GitLab or whatever... and at
merge time, we would rewrite their history to remove the .deb, and
import it into our APT repo.