Commit 382b76a4 authored by 127.0.0.1's avatar 127.0.0.1 Committed by amnesia
Browse files

text

This reverts commit 6ae6ac07
parent c1bd4c35
......@@ -173,328 +173,3 @@ happiness, and a warm feeling of being part of a broader community.
Another aspect is that we would essentially stop having to produce and
maintain backports of Debian packages.
<a id="rolling-stretch-analysis"></a>
## Analysis of the Stretch cycle
### Statistics of Git activity
We have generated
[statistics and graphs](https://labs.riseup.net/code/attachments/download/1640/gitstats-2.x-to-3.0.tar.bz2)
of the Git activity between our Jessie-based branch and our
Stretch-based one. Long story short:
* We really started this work in 2016-08.
* The first peeks of activity are the two first sprints we had in
2016-08 and 2016-11. Then quite some work was done every month
until the release, with more sustained activity starting in
2017-03.
* The biggest peaks are:
- 2017-03: we had a Stretch + reproducible builds sprint, and the
work on reproducible builds is accounted for in these stats (it
was based on Stretch).
- 2017-05: we had a Stretch sprint and our technical writers were
busy updating the doc.
### Code
XXX:anonym, please add whatever input data, feelings and analysis
you can come up with.
Commits history in the `auto` and `config` directories between our
Jessie and Stretch -based branches:
- 2016-01: 17
- 2016-02: 1
- 2016-03: 0
- 2016-04: 0
- 2016-05: 14
- 2016-06: 0
- 2016-07: 16
- 2016-08: 59 (in-person sprint)
- 2016-09: 2
- 2016-10: 0
- 2016-11: 88 (in-person sprint)
- 2016-12: 35 (remote sprint)
- 2017-01: 24 (remote sprint)
- 2017-02: 10 (remote sprint)
- 2017-03: 108 (in-person sprint + lots of work on reproducible builds)
- 2017-04: 27
- 2017-05: 97 (in-person sprint)
- 2017-06: 42
So about 2/3 of the commits were done during 4 in-person sprints,
which was the plan. Interestingly, remote sprints were less
productive, but let's take this metric with a grain of salt.
Clocking data:
* intrigeri worked 44 days on the Stretch migration. Sadly, this
includes some work that's irrelevant here, like:
* Some test suite work
* Porting to the new Greeter
* New memory wipe implementation
* Preparing 5 alpha/beta releases, most of which we wouldn't
have needed if we had been tracking Debian testing.
* Reviewing other contributors' work targeted at Stretch, which
would be instead part of RM'ing if we were based on
Debian testing.
* anonym mostly worked on the test suite, which doesn't count in this
section. Let's assume the code work he did somewhat compensates the
time erroneously clocked by intrigeri.
* A few other people also did some work, which was helpful, but
that's negligible vs. the total time spent.
So all in all, we can assume about 44 days of work in this area.
Dividing this number by the number of 3-months cycles in a Debian
release cycle (8), that's 5.5 days every 3 months. But of course some
work would have to be done more than once in two years if we tracked
Debian testing, so this figure probably reflects an unrealistic best
case scenario.
Regarding code changes we had to do to adjust for changes in testing:
Stretch was frozen for half of the experiment, so we haven't much data
here. After looking closely at the Git log, what I could spot is
mostly unfuzzying patches, cherry-picking packages from sid when they
were auto-removed from testing, and rebasing packages we patch on top
of the current version, i.e. trivial (although somewhat boring) work
whose amount we could substantially lower on the long term by working
more closely with upstream projects; the only bigger pieces I could
spot are the migration to GnuPG v2 and Icedove → Thunderbird, but
those would have to be made exactly once as well if we were based on
Debian testing.
Regarding tracking security issues fixed in sid but not in testing:
* blocked due to the normal transition time from unstable to testing:
we didn't pay attention to this when releasing Tails 3.0~, so we
have no data here; too bad.
* blocked by some ongoing transition: we started providing security
support for Tails 3.0~ after Stretch was frozen, so we had no
direct experience of this problem; too bad.
### Test suite
XXX:anonym, please add whatever input data, feelings and analysis
you can come up with.
Non-merge commits in `features/` and `run_test_suite` between our
Jessie-based branch and our Stretch-based one:
- 2016-05: 1
- 2016-06: 0
- 2016-07: 1
- 2016-08: 62
- 2016-09: 0
- 2016-10: 0
- 2016-11: 41
- 2016-12: 39
- 2017-01: 7
- 2017-02: 30
- 2017-03: 29
- 2017-04: 19
- 2017-05: 20
Here as well, most of the work was done during a few sprints.
Clocking data:
* intrigeri didn't clock his test suite work separately from his
other Stretch work, but that probably wasn't more than 2-4 days.
* anonym worked 24 days during sprints + XXX days outside of sprints
on the Stretch migration; most of this time was spent on the
test suite.
Total diff stat:
- 258 files changed, 1252 insertions(+), 986 deletions(-)
- among those, 200 images changed: 52 removed, 13 added, 135 updated;
and among these 200 image changes, at least 50 are orthogonal to
what we're analyzing here:
* many of the removals are thanks to dotail-ification, so at least
those won't need to be updated ever again
* 21 due to switching to Tor Browser 7.0
* 1 due to the memory wipe system change
* 23 due to the new Greeter
Now let's do some stats ignoring the changes that are orthogonal to
what we're analyzing here, in that we would have to do them once,
regardless of what version of Debian (stable or testing) we're
tracking, i.e. the Icedove → Thunderbird rename and the memory erasure
system replacement:
- `*.feature`: 19 files changed, 152 insertions(+), 125 deletions(-);
most of the changes are still orthogonal to what we're analyzing
here (refactoring and a few new regressions tests for bugs that
affect Tails 2.x too)
- `*.rb`: 21 files changed, 661 insertions(+), 502 deletions(-); many
of the changes are still orthogonal to what we're analyzing here
(refactoring and porting to the new Greeter and memory erasure
system)
In short, by far the biggest part of the porting work was done by
updating images and dogtail-ifying tests. Only very few other bits had
to be updated (GnuPG v2, migrating away from obsolete CLI, adjusting
to new `nmcli`, this kind of things).
Now, of course we can't ignore the dogtail-ification work when we
think ahead about how test suite maintenance would look like if we
tracked Debian testing:
* if we hadn't done this work, we would have lots more images
updates, and way fewer image deletions;
* it's mostly one-time work, so what's been done is done for good,
still we should keep investing in it and dogtail-ify more tests in
order to decrease the amount of images we need to maintain. So at
least for a few 3-months cycles, this work will be recurring.
Additional data that would be interesting:
* Did we have to update some tests (e.g. images) more than once due
to changes in Debian testing?
### Documentation
XXX:sajolida, please add whatever input data, feelings and analysis
you can come up with.
Note: the following stats ignore PO files, the Icedove → Thunderbird
renaming, and the `blueprint`, `contribute`, `inc` and `news`
directories. Sorry I realized too late that I should have taken the
3.0 release notes into account :/
Non-merge commits in `wiki/src` between our Jessie-based branch and our
Stretch-based one:
- 2016-08: 1
- 2016-09: 0
- 2016-10: 0
- 2016-11: 2
- 2016-12: 0
- 2017-01: 3
- 2017-02: 2
- 2017-03: 42
- 2017-04: 26
- 2017-05: 45
- 2017-06: 1
So this confirms what we already knew: the bulk of the work was done
in the last 3 months before the release, and started after Stretch was
already deeply frozen, so obviously we won't learn much here regarding
_continuously_ updating our doc to match a moving target.
But at least we can get an idea of the total amount of work this has
required :)
- 84 files changed, 417 insertions(+), 355 deletions(-)
- Among the 84 changed files: 35 pictures + 49 text files.
Some of these changes are orthogonal to what we're analyzing here, in
that we would have to do them once regardless of what version of
Debian (stable or testing) we're tracking:
- New Greeter: 8 files changed, 151 insertions(+), 21 deletions(-)
- Most of the changes in the `install` directory are either about the
move to 64-bit, or about supporting installation from Debian
Stretch, or about the new Greteer as well: 21 files changed, 81
insertions(+), 100 deletions(-)
- KeePassX migration: even if we had been tracking testing, this
would have had to be done only once: 1 file changed, 64
insertions(+), 29 deletions(-)
So very roughly, the changes that would be impacted by the "let's go
rolling?" decision are closer to 54 files changed (24 pictures and 30
text files), 121 insertions, 205 deletions.
So, the worst case scenario is that every 3 months we have to:
* go through changes in the packages list, and identify what doc
pages and screenshots need updates; this can be semi-automated;
* update about 24 pictures
* update 100-200 lines in about 30 text files
Let's keep in mind some other factors that will make reality a bit
nicer:
* There's a GNOME release only every 6 months, so a non-negligible
part of our doc will have to be updated only twice a year, instead
of every 3 months. I bet many of our screenshots are part of
this lot.
* Our test suite can help identify the doc bits that need an update:
either text-based instructions (when our tests use dogtail), or
screenshots (when our tests use picture recognition). There are
a few ways we could boost this beneficial factor as we go e.g.:
- add automated tests specifically about things we document;
- add steps to our existing tests to validate the screenshots we
have in our documentation.
Clocking data: XXX (ideally, specify clearly what's accounted for,
e.g. whether the work listed above as irrelevant here is).
### Summary
Looking at the data we have, my (intrigeri) general feeling is rather
positive but I was sold to this idea from the beginning, so let's be
clear: I'm biased :)
Now, I feel we haven't gathered enough data during the Stretch cycle
to make a final decision wrt. starting to track Buster in 2018-01 or
not: particularly in the doc and security support areas, we have very
little data about _continuously_ adapting to a moving target.
But I think we do have enough data to decide at the summit something
like "if X happens by the end of November and takes us less than
Y days, then we'll go ahead and switch to Buster in 2018-01".
Worst case we'll suffer for a year until Buster is frozen, and then we
won't try again before fixing some pre-conditions we'll have
identified during our failed first try.
I think the only way we can gather the missing data, and check if "X
happens" and whether it takes "less than Y days", is to have sprints
in 2017-10 or 2017-11, and actually port most of our code, test suite
and documentation to Buster. We could take some shortcuts though, e.g.
by estimating the amount work needed instead of doing it, when we feel
very confident we can come up with a realistic estimate.
It'll be impossible to have all these kinds of work done in parallel
during one single sprint though; the best process I can think of is to
serialize the tasks like this:
1. 3 days: make the ISO build and boot (anonym + intrigeri)
2. 3 days: loop over "port some of the test suite, fix regressions in
our code to make the ported tests pass" (anonym + intrigeri)
3. 1 day: give our technical writers a list of software that has
changed in a way that may affect our documentation, so they can
update it (anonym + intrigeri)
4. 3 days: update the doc (sajolida & his amazing tech writing band)
If we eventually decide to postpone our migration to Stretch, some of
this work will be wasted, and hopefully some won't. To minimize the
amount of wasted work in the worst case, we should be ready to give up
in the middle of the afore-described process, if we notice it's going
to take us more time than "Y days".
While doing this we should carefully gather the data we did not get
during the Stretch cycle, in particular:
1. How would we address security issues fixed in sid but not in
testing? (see the "Stretch cycle" section for details)
2. How can developers convey to technical writers what changes may
affect the documentation? (see ideas in the "Documentation" section
above)
3. Thanks to #2, can we identify what piece of doc needs updating
_without_ having to test the entire documentation every
three months?
When deciding what X and Y should be, and when analyzing how the
experiment went, we shall keep in mind that Debian testing changes _a
lot_ when the gates open after the freeze. It's not exactly the time
when it's the most reliable, even though there's been big progress in
this area. And there are lots of library transitions going on, that
could be painful for us, even though there's been big progress here as
well. In other words, the delta between Stretch and "Buster in
October" will be substantially bigger than what will happen during the
next 3-months cycles, and the porting work will be close to the
maximum one can expect during these 3-months cycles.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment